Avaya ERS Switch/Stack Monitoring Through SNMP

[bylie]

Hi,

Because it's a quiet period at the office I'm taking the time to get some smaller projects done that were already on my todo list for too long ;-). One of them is to write a Nagios plugin which will monitor various aspects of our Avaya ERS 25xx/45xx/55xx switches in the field. The idea is to be able to proactively act on events such as fanfailures, high temperatures, ... which might go unnoticed at first until things go from bad to worse...

So far I've found the following interesting OID's to monitor:

envFans: 1.3.6.1.4.1.45.1.6.3.3.1.1.10.6
envTemp: 1.3.6.1.4.1.45.1.6.3.7.1.1.5.5
cpuLast10Seconds: 1.3.6.1.4.1.45.1.6.3.8.1.1.11.3
cpuLast1Minute: 1.3.6.1.4.1.45.1.6.3.8.1.1.5.3
cpuLast10Minutes: 1.3.6.1.4.1.45.1.6.3.8.1.1.6.3
cpuLast1Hour: 1.3.6.1.4.1.45.1.6.3.8.1.1.7.3
cpuLast24Hours: 1.3.6.1.4.1.45.1.6.3.8.1.1.8.3
cpuTotal: 1.3.6.1.4.1.45.1.6.3.8.1.1.4.3
memTotal: 1.3.6.1.4.1.45.1.6.3.8.1.1.12.3
memAvail: 1.3.6.1.4.1.45.1.6.3.8.1.1.13.3
ifPortStatus: 1.3.6.1.2.1.2.2.1.8.<portindexnumber>

I'm really not an SNMP/MIB expert here but does anyone have any other interesting OID's which would be worth monitoring? I'm actually still looking for some way to check the health of a stack or the status of the stackingports but I can't seem to find an OID which would give me this information.

[Michael McNamara]

Thanks for sharing the information @bylie.

You can setup a new feature called "Stack Monitor" on the stackable Ethernet Routing Switches and the switch will issue an SNMP trap when the number of expected switches/units in the stack drops below the configured value. Then you just need to configure your management workstation to page or email you when that happens. I'm using currently HP OpenView so it just sends the team an email to let them know that we need to check the stack for an issue.

Cheers!

[bylie]

Unfortunately we currently don't have a trapreceiver like HP OpenView or OpenNMS. We're going to use COM however (as replacement for ESM) which should also be able to receive SNMP traps. Does anyone know if COM is also capable of sending emails based on information of these traps?

Maybe another idea: does anyone know how (by which OID) JDM gets the state of the status LEDS shown in the physical view of the switch? I was thinking that maybe if I can get/walk these through SNMP I can easily create some code to verify the stackhealth and maybe even identify single stackinglink failures, just an idea...

[Michael McNamara]

If you have a Linux host available you could use Net-SNMP's trapd receiver along with SNMP Translator. You don't need a commercial solution to setup a trap receiver and have it perform simple notifications or even attempt to automatically gather additional diagnostic information by launching additional scripts or applications.

http://sourceforge.net/projects/snmptt/
http://sourceforge.net/projects/simple-evcorr/

Good Luck!

[Michael McNamara]

I almost forget one of the best resources;

http://www.snmplink.org/snmpresource/trap/

[bylie]

Thanks for the tips I may have a look at SNMPTT in combination with our Nagios install.

I was however already looking through the MIB's and found this entry which might be interesting:

s5ChasBrdLeds 1.3.6.1.4.1.45.1.6.3.4.1.1.2

This should give the frontpanel led's status of a single switch or of each unit in a stack. I'm only having a bit of trouble interpreting this. According to the MIB it should be arranged like this:

Code: [Select]

A bit array that contains the value of the
front panel LEDs and the annunciator. This is
a packed bit string; each LED is encoded into
4 bits (a nibble). The first nibble is stored
in the high order bits of the first octet. The
second nibble is stored in the low order bits
the first octet, etc. However, the current
mapping is that the first three nibbles are
reserved, the fourth nibble has the value of
the 'annunciator', and the 'normal' LEDs start
with the fifth nibble. If the LED values are not
available, a zero length string is returned. The
following shows the meaning of each bit (bit 0 is
the least significant bit) when it has a value
of one:

  bit  meaning
  ---  -------
   0   green light
   1   yellow light
   2   blinking
   3   reserved
I'm getting a single or multiple (for a stack) hex string(s) back from the snmpwalk but even when I convert to binary it's still not clear which bits would represent the status of the up/down stackingports. I'm guessing that this is really giving back the status of all the led's on the frontpanel which also include all the ports...

[bylie]

Hate to reply to my own post but I'm a bit stuck with this. When I do a snmpwalk of the s5ChasBrdLeds I get the following result for a 2 x 2526T stack:

Code: [Select]

snmpwalk -v 2c -c public <ip> 1.3.6.1.4.1.45.1.6.3.4.1.1.2
SNMPv2-SMI::enterprises.45.1.6.3.4.1.1.2.10 = Hex-STRING: 00 00 11 51 01 00 00 00 05 00 00 00 00 00 00 00
00 05 00 00 00 00 00 00 00 00 00 00 00 05 00 00
00 00 00 01 00 00 00 00 00
SNMPv2-SMI::enterprises.45.1.6.3.4.1.1.2.20 = Hex-STRING: 00 00 11 51 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 55 55 00 00 00 00 00 00 05 00 00
00 00 00 01 00 00 00 00 00
But I can't seem to find how this translates to the attached led layout in JDM. Maybe I'm just looking in the wrong place but this has got me intrigued...

[Michael McNamara]

There's some conversion that needs to go here since the Net-SNMP toolset is returning you a hex value and not a binary string.

Here's what I get from a standalone ERS 4548 with only the power and status lights green;

SNMPv2-SMI::enterprises.45.1.6.3.4.1.1.2.10 = Hex-STRING: 00 00 10 50 00 20 00 00 02 00 20 50 00 00 05 00
50 00 00 10 10 10 00 00 00 50 50 50 00 00 00 00
00 00 01 00 00 00 00 00 55 00 20 00 12 00 00 00
50 00 55 00 00

You need to convert the hex string in a binary string and then apply the logic you posted above from the MIB.

I'm honestly to tired right now to-do the math but it shouldn't be hard to figure out.. with the binary you can perform a AND or XOR with a binary bitset and see if that bit is set or not.

Have a look at this article for some background; http://en.wikipedia.org/wiki/Bit_array

Good Luck!

[bylie]

I've converted hex to binary and this is what I got so far (I think I'm getting somewhere):

Code: [Select]

Unit1:

00000000 00000000 00010001 01010001 00000001

00000000 00000000 00000000 00000101 00000000 00000000

00000000 00000000 00000000 00000000 00000000 00000000

00000101 00000000 00000000 00000000 00000000 00000000

00000000 00000000 00000000 00000000 00000000 00000000

00000101 00000000 00000000 00000000 00000000 00000000
00000001 00000000 00000000 00000000 00000000 00000000

Unit2:

00000000 00000000 00010001 01010001 00000000

00000000 00000000 00000000 00000000 00000000 00000000

00000000 00000000 00000000 00000000 00000000 00000000

00000000 00000000 00000000 00000000 01010101 01010101

00000000 00000000 00000000 00000000 00000000 00000000

00000101 00000000 00000000 00000000 00000000 00000000
00000001 00000000 00000000 00000000 00000000 00000000

• Nibble 6 and 8 seem to be for the stacking led's. I do still have to test which is which however. Nibble 10 seems to be for the base led. I kind of deduct this because all of these do not show up when I query a standalone switch instead of a stack.
• Nibble 5 might be for the power led I think and nibble 7 might be for the status led although it's strange that this indicates that it's blinking...
• Nibble 11 through 22 are for the activity leds from port 1 to 12 and then there seems to always be 12 nibbles which are not used on the ERS 2500 series (they are physically not there) but which probably are used on the 4500 series as activity leds while the former leds are then used as portspeed and PoE indicators. This repeats itself for all the available ports. The shared SFP ports seem to utilize nibble 59/71 and 60/72 to display their status (on an ERS 2526T that is) where 71 and 72 are the speed indicators and 59 and 60 the indicators for activity. This all shifts up when something like a ERS 2550T is used ofcourse.

So to know the status of the stackingports, as indicated by the frontpanel led's, we need to look at:

00 00 11 51 01 ...

And maybe also the base status might be interesting to know:

00 00 11 51 01 ...

[jbfuzier]

I wrote this perl script for Nagios, it works well with our ERS 4500 and 5000...

[bylie]

I wrote this perl script for Nagios, it works well with our ERS 4500 and 5000...

Thanks a lot for the script! It essentially does the same as what I wanted to do for our switches. The problem is I'm not really a Perlguy, I was initially looking to write my own using Python but I'll have a look to see if I can understand it and maybe I can then just extend the script to also gather the statusleds for extra information.

Next to this I'm probably also going to have a look at how much extra configwork it would be to setup Nagios as a SNMP traphandler, it should be possible (according to some sources) but it never really was the primary goal of the software so it feels a bit bolted on. Something like HP OpenView or OpenNMS would probably suit this role better but that would mean setting up and maintaining yet another system which also has it's downsides.

[jbfuzier]

I am not really a perl guy either I didn't have much time to write this so it is really a quick and dirty script, it can be improved a lot...

[bylie]

I've put together a Pythonscript which now checks all of our ES 470, ERS 2500 and ERS 4500 switches/stacks in Nagios for the following parameters:

• Average CPU usage over the last 10 minutes. I guess this might be interesting to indicate when the CPU might be choking on something especially when starting to use more advance features (EDM, DHCP snooping, DAI, ...).
• Free memory. This might be interesting when starting to use more advanced features which might be more demanding and maybe even to indicate things like memory leaks in new firmware releases slowly consuming all memory.
• Fan status
• Temperature if the switch/firmware supports it.
• Chassis led's to do a quick sanitycheck of the stackingports, basestatus, ...

All of these checks are discovered automatically when the script is pointed to a switch/stack IP in order to deal with devices which might not support all of them because of either older firmware or just not being supported. The output is the standard Nagios OK, Warning, Critical or Unknown exitcodes when any of the checks might encounter something fishy, so in fact these are not individual checks but all of them are always run together to give a general indication of the health of the device.

In the end I've done the scripting in Python myself because somehow Perl and I will probably never be friends and I've been using Python on and off through the years at work and for my own personal scripting needs.

I'll probably put the script up here after some final cleaning/polishing and testing but first I'm also going to write a second version to check the KHI OID's of our new VSP9000 to keep an eye on her .

[bylie]

So well I've gotten my first alert from my new check, during the weekend ofcourse , and it seems to be... a false positive:

Critical - Temperature:(unit 1 critical 80.5°C)
OK - CPU:(unit 1 OK 7% used) Memory:(unit 1 OK 66% used) Fans:(unit 1 fan 0 OK, unit 1 fan 1 OK, unit 1 fan 2 OK, unit 1 fan 3 OK) LED's:(unit 1 OK)

This is a standalone ERS 4524GT switch running FW:5.3.0.3 and SW:5.3.3.011. The check has been running for about 4 days without any problems and other temperature sensors in the same datacenter indicate a normal temperature. So unless the switch has caught fire or something I'm presuming this is a false positive and maybe a bug (in the readout or the SNMP code) or maybe the temperature sensor has gone haywire (which would be quite a coincidence after just starting to do these checks). Anyone else seen something like this? I've already gone through the releasenotes of all the new releases on the Avaya website but have not seen any mentioning of this being a resolved or known issue.

There's a picture attached which verifies that JDM indicates the same which is normal I guess because it's querying the same SNMP OID.

The CLI also seems to agree that the temperature is high:

Code: [Select]

(config)#show environmental
Unit# PSU1      PSU2      FAN1 FAN2 FAN3 FAN4 Temperature
----- --------- --------- ---- ---- ---- ---- -----------
1     Primary   N/A       OK   OK   OK   OK   HIGH 80.5C
A "show logging sort-reverse" doesn't show any entries about this problem however... no traps or entries whatsoever.

[Slava N]

this problem solved in release 5.5. after tests in release before 5.5 switch temperature indicated 80.5 C and more.

Code: [Select]

TEST542#show environmental
Unit# PSU1      PSU2      FAN1 FAN2 FAN3 FAN4 Temperature
----- --------- --------- ---- ---- ---- ---- -----------
1     Primary   N/A       FAIL FAIL FAIL FAIL HIGH 50583652C