Monitoring Nortel ERS equipment

[stauftm]

Just wondering what people are doing out there to monitor your Nortel ERS equipment? I currently use MRTG to monitor my network bandwidth (See Attached) and basic syslog for alerts, but wondering if anyone is doing anything more? Also the value in what you are monitoring. The MRTG has saved me many times, especially on our Internet Uplink port (5Mb). I can use the information for justifying more speed, current load, etc.

Regards,
Todd

[Michael McNamara]

Hi Todd,

I've been using MRTG/RRD for years and can't say enough great things about it. I started to play with Cacti a few months back but other work pulled me away and I haven't gotten a chance to finish my evaluation. I've actually been using 14all.cgi with some custom changes and have it integrated into my corporate Intranet site. It works really nice, I can just place a URL in the CSM solution and up pops a dynamically generated graph.

We also run a custom written Perl script that calls fping and pings (ICMP) all 700+ devices we monitor every 60 seconds. This insures that we know immediately if there is a problem with any of the devices. On top of that I would probably tell you that SNMP traps are the next most important component to any network management. SYSLOG is great but unless you have a script like swatch examining the syslog data you may miss some important events. I use HP OpenView but you could definitely get away with utilizing Net-SNMP's trap daemon configuring it to take various specific actions when a specific SNMP trap is received.

I currently generate email messages for the following SNMP traps, power supply failure, card failure, SLPP down, VLACP down, VLACP up, SMLT down, SMLT up, temperature alarm, UPS on battery, UPS off battery, IST up, IST down, etc.

Have a look at the graph below for a very busy 50Mbps Internet connection.

Good Luck!

[qazzie]

Well I won't deny that MRTG is not ok. But I really dislike the 5 minute interval. I use PRTG a lot more often. You can poll every 10 seconds and in case for cpu monitoring a lot better, since MRTG vagues out CPU spikes. And the bundeling of links (mlt's) etc I found a lot easier.

[Michael McNamara]

There are a lot of very interesting and compelling solutions out there, and PRTG is certainly one of the better known and more respected.

You can run MRTG/RRD/14all.cgi with 1 minute intervals. I'm doing that for a few interfaces and SNMP OIDs (CPU Utilization) but when you have 600+ switches to monitor and some 6,300 interfaces to graph you really can't be polling them all every 60 seconds.

I also played with Nagios a few years back when it was called NetSaint and was very impressed. Since that time there have been a lot of start-ups and emerging products. SmokePing is another tool from Tobias Oetiker, author of MRTG, that might also be worth a look if your interested in graphing your network's latency. Zenoss is also another option for the larger network management type of solution.

I'm going to stop now because the list is nearly endless... I'll let a few of the other folks chime in with their favorites.

[rjenk]

I am a big fan of Solarwinds products.  We started with their Toolset and migrated to Orion.  We use it for statistical information, logging, and alerting.  Support has been very good and they have a fairly good networking forum as well.

http://www.solarwinds.com
http://www.solarwinds.com/products/orion/

[modestgeek]

I use a combination of the following open source tools.

Nagios 3.x for up/down of services/devices which I'm moving to the Icinga fork. http://www.icinga.org/
Cacti for interface traffic graphing http://www.cacti.net/
Netdisco for network map generation/switch port mapping http://netdisco.org/
IPPlan for IP/DNS address management http://iptrack.sourceforge.net/

I'm looking to replace both Netdisco and IPPlan with NetDot which is being developed out of the University of Oregon. It's going to be amazing once it's complete. Track IP address, cable plant, switch port mapping, etc. https://netdot.uoregon.edu/trac/

[topendharness]

With our new project deployment of a Small Campus Nortel network, we are going to use Zenoss.  It has been a good product for us in the past with a great support community as well.  Easy to use, the free Zenoss Core version is very capable.
Check it out.

[TheKingSlacker]

I would also recommend Solarwinds Orin (NPM).. I just recently setup Solarwinds to monitor and notify the health of my switches.. Out of the roughly 400 nortel switches we had discovered 27 switches with 1 or more failed fans and a couple of units with bad PS (which were still up thankfully because they add RPSes)

I would also recommend Solarwinds Kiwi CatTools for backing up switch configuration.. Single license is only $500

The King.

[pgoggins]

We've been using Cacti for a number of years. There are some addon's available such as CAMM (syslog reporting) and Monitor (What's Up Gold dashboard) view. Cacti can be used for up/down monitoring by changing the query interval to every 60 seconds instead of 5 minutes......I would not suggest adjusting the graphs to store data in shorter than 5 minute intervals.

[DaveAnderson]

Michael - your posts have been very insightful.  Can you provide more info on the SNMP MIB values used to monitor the VLACP Up and VLACP Down?

Thanks

Dave

[Pollay]

Michael - your posts have been very insightful.  Can you provide more info on the SNMP MIB values used to monitor the VLACP Up and VLACP Down?

Thanks

Dave

From rc-trap-mib:

rcnVlacpPortDown NOTIFICATION-TYPE
    OBJECTS       { rcPortIndex}
    STATUS        current
    DESCRIPTION  "A rcnVlacpPortDown trap signifies that
                  Vlacp is down on the port specified."
    ::= { rcTrapsMib 80 }

rcnVlacpPortUp NOTIFICATION-TYPE
    OBJECTS       { rcPortIndex}
    STATUS        current
    DESCRIPTION  "A rcnVlacpPortUp trap signifies that
                 Vlacp is Up on the port specified."
    ::= { rcTrapsMib 81 }

Numerical oid values are 1.3.6.1.4.1.2272.1.21.80 for down and 1.3.6.1.4.1.2272.1.21.81 for up

[TankII]

Plixer tools
Syslog/Eventlog/SNMP = Logalot
MRTG based graphing = Denika
Netflow/IPFix = Scrutinizer

Denika has the option of a real-time graph if you are trying to watch at the time of a problem.  1 Second intervals.
Custom templates for CPU utilization, Memory Utilization, Fabric utilization, and CPU/FAN inlet temperature on our 8600's, 5600's, 5500's and 470's.

WhatUP Gold for the NOC.

TankII

[Dominik]

Hi,

there are alot of good tools out in the wild and I think most of them can be used with the network equipment from Avaya.

I used a long time Cacti, HP OpenView and the EnterpriseSwitchManaher (ESM).

At the moment I would recommand to use the following tool set:
-QRadar (QLogic IBM) for IPFIX Flows and Syslog
-COM (AVAYA) for configurations and traps
-OMD (Open Monitoring Distribution) for monitoring. OMD is a collection of opensource tools that can be
 installed with a single packet. It contains:
     * nagios-3.2.3
          o nagios-plugins
          o nsca
          o check_nrpe
    * Shinken-0.6.99
    * nagvis
    * pnp4nagios
    * rrdtool/rrdcached
    * Check_MK
    * MK Livestatus
    * Multisite
    * dokuwiki
    * Thruk
    * Mod-Gearman
    * check_logfiles
    * check_oracle_health
    * check_mysql_health
    * jmx4perl
    * check_webinject
    * check_multi
  The very impressing thing about OMD is that all is integrated and you donīt have to setup a lot of   
  parallel working instances of monitoring, all is unified in one system.

Cheers

[DaveAnderson]

You guys rock!!!  Here is additional info I got from an open ticket with Nortel (now Avaya)...

*********************************************
Regarding your question, there is a variable you can monitor via SNMP in order to get the VLACP status of a port.
It is not part of standard variables, rather included in the proprietary MIB, it is called rcPortVLacpPortState.
The full OID for it is: 1.3.6.1.4.1.2272.1.4.10.5.1.9.

It returns an integer value with the following meaning : up(1), down(2)

You can calculate the value for ifIndex with the formula below:
ifIndex = (64 * slot) + (port - 1)

Example:
ifIndex of 4/48 = (64 * 4) + (48 - 1) = 303

*********************************************

Thanks for the help guys.  I was able to script up a custom monitor in our SNMP monitoring system and found that VLACP went down again this weekend.  The carrier has been having issues and we have been seeing up Interfaces, but unable to pass traffic over the circuit.

[Michael McNamara]

Hi Dave,

I'm happy to hear you figured it out on your own.

I'm not sure if you tried searching the forums but you'll find a lot of information here.

http://forums.networkinfrastructure.info/scripting/ifindex/

Cheers!