Packet Discards

[bwilliams2]

How do you deal with packet discards?

Our 4548's have a lot of packet discards. Over 500,000 a day. I am not sure where to start to trace these down.

Thanks

[bylie]

What specific metric are you referring to, "InDiscards" or "NoResourcesPktsDropped"?

• "InDiscards", from my experience, are almost always caused by a port that is receiving tagged frames for a VLANID that that port is not a member of. If I'm not mistaken, in the past, a switch would just put these frames onto the PVID VLAN causing frames to be bridged to another VLAN posing a possible securityissue. But with the more recent "FilterUnregisteredFrames" option, which is on by default, those frames are now rightfully discarded. So it probably is the result of some "lazy admins" not cleaning up both sides properly.
• "NoResourcesPktsDropped" on the other hand are generally caused by a switch that's "low on/out of" buffer memory, for that particular group of ports, so it will start dropping packets. I've seen the absolute value of this metric increasing on some of our ERS 2500 and ERS 4500 ports and I've actually talked about this with our local Avaya representative and they state it's not really a problem or it might just be a bug. We haven't had any complaints from users though so I'm not really investigating it further at the moment.

[Michael McNamara]

• "InDiscards", from my experience, are almost always caused by a port that is receiving tagged frames for a VLANID that that port is not a member of. If I'm not mistaken, in the past, a switch would just put these frames onto the PVID VLAN causing frames to be bridged to another VLAN posing a possible securityissue. But with the more recent "FilterUnregisteredFrames" option, which is on by default, those frames are now rightfully discarded. So it probably is the result of some "lazy admins" not cleaning up both sides properly.

As bylie mentions above, the vast majority of InDiscards will be from VLANs being improperly tagged across uplinks/downlinks.

Cheers!

[bwilliams2]

VLANS not on both sides was the culprit. Thanks for the help fellas.

[ChesTrulson]

I'm digging up an old thread here, but does anyone have any info about OutDiscards and NoResourcesPktsDropped?

I have as many as 440 MILLION of these on some of my 5530 ports, i'm now discovering.

One stack in particular is giving me grief, seems to be dropping a couple of dozen frames about once every minute.  Oddly, the switch is not heavily loaded.  There is an MLT uplink to an 8600 on 1/24,2/24, with about 500mbits/s in across it, and an MLT out on 1/1,2/1 to a blade enclosure that's pushing about 500mbits/s out.

Can anyone explain this?

[Flintstone]

Hi,

'NoResourcesPktsDropped' is usually seen when the port buffers are over run with data.  In my case this was due to a Network loop.

'OutDiscards' on my ERS5560's is the same value as 'InDiscards', so I have put this down to mis-matched Vlan tagging in the past?  I would check your Vlan configurations on both ends of the tagged port?

CheerZ and goodluck

[ChesTrulson]

That's the thing, i don't think that there's enough traffic on the switch to have filled the buffers...

Has anyone seen a block diagram for these switches?  Does this 'buffer' that's full serve a single port, or a group?

[Michael McNamara]

The buffer size allocation (per port) is configurable... let me dig it up... somewhere here...

http://support.avaya.com/css/P8/documents/100123718

See page 15 for the following reference;

4. Queue Sets
Prior to software release 4.0, the Ethernet Routing Switch 5500 supported a single queue set with eight queues, one absolute queue and seven WRR queues.
With the introduction of software release 4.0, eight different queue sets where made available. Each queue set has different characteristics in regards to number of queues and service weights allowing the user to select a queue set based on the user‟s particular needs. With eight queue settings and three resource sharing options, the Ethernet Routing Switch 5500 supports a total of 24 different queues and buffer setting combinations. Prior to making any changes to the egress queue, the buffer resource sharing feature must be enabled.
Resource Sharing
The three (3) possible resource sharing settings in version 4.0 or greater software release are regular, large, and maximum. These settings allow the user to change the amount of buffer which can be allocated or shared to any port. Note that the switch must be rebooted if any changes are made.
Table 4: Ethernet Routing Switch 5500 Resource Sharing Setting Description
Regular
1 port may use up to 16% of the buffers for a group of 12 ports.
Large
1 port may use up to 33% of the buffers for a group of 12 ports.
Maximum
1 port may use 100% of the buffers for a group of 12 ports.

Cheers!

[ChesTrulson]

Perfect!  Thank you!

This switch is set to 'Large' currently, i'll change it to 'Maximum'.  It makes perfect sense that this is causing drops, as there is only 1 busy port on each of the 12 port groups.  I'll let you all know if it fixes the problem.  It's a pity you need to reboot the stack to apply the change

I guess i'll have to have a look around at my other data centre switches and see if i need to make this change on them as well.

Also from that doco, some recommendations on which size to use:

Resource Sharing Commands
 5520-24T-PWR(config)# qos agent buffer <large | maximum | regular>
The qos agent buffer <regular | large | maximum > command allows the user to specify the level
of resource sharing on the switch.  This parameter is global and requires a reset to activate a
change.  This command is in the CLI priv-exec mode.

 5520-24T-PWR(config)# default qos agent buffer
The default qos agent buffer command sets the switches agent buffer back to a default setting of
regular.  In order for this command to take affect, a reset of the switch must occur.  This
command is in the CLI priv-exec mode.

Resource Sharing Recommendations

Avaya recommends you use the default resource-sharing setting of regular.  If you
change the setting, the resulting performance may increase for some ports, and at
times, decrease for other ports.
Generally speaking, smaller buffers achieve lower latency (RTT) but reduce the throughput ability which is
better for VoIP etc. and sensible jitter application. 
You should use the Maximum resource sharing setting:Filt
Filters and QOS Configuration for Ethernet Routing Switch 5500
Technical Configuration Guide
October 2010 16
avaya.com

 If you are using your 5520 for big file transfers (like backup of servers)

 If you are using (the AppleTalk Filing Protocol) AFP, use large or maximum resource sharing
(AFP use a fix windows size set to 65,535K).You should use the large resource sharing setting:

 If you are using your 5520 for high bandwidth application such as video.

 If you are using large TCP windows for your traffic, use large resource sharing (you can also
reduce the TCP windows size on windows operating system - see Microsoft TechNet article
224829).

 If you have 4 or fewer ports connected per group of 12 ports.
You should use the Regular resource sharing setting:

 If you are using your 5520 in a VOIP environment.

 If you have 5 or more ports connected per group of 12 ports

[Dominik]

Well; the buffer size allocation is one of these ugly thinks, that makes me sometimes a headache.

If you have only one high traffic port in the portgroup of 12 ports you will be able to fix the packet drops on no ressources with
changing the value to maximum.
On siwthces with more high traffic port than one you can split these ports to the portgroups. On a 5510/24t this will be 2.
And thats it what you can do.

bye

[Flintstone]

Hi,

It might also be an idea if you can trace the root cause of the problem.  Why is the traffic over running the switch buffers? 

It sounds like you might have devices that are unable to receive traffic at the required senders rate?

CheerZ

[bylie]

Hi,

It might also be an idea if you can trace the root cause of the problem.  Why is the traffic over running the switch buffers? 

It sounds like you might have devices that are unable to receive traffic at the required senders rate?

CheerZ

I'm not sure if I understand all of this correctly but than doensn't a device that's not capable of receiving packets at a certain speed pose a problem to the entire portgroup utilizing this buffer when the qos agent buffer is set to maximum? Because once the buffer is full no other port of the portgroup has any bufferspace left? I can imagine TCP will control the flow of packets making use of all sorts of flow control and/or congestion algorithms but what about UDP for example which would just keep sending? Would something like layer 2 flow control help here?

I've also investigated the "NoResourcesPktsDropped" a bit and have seen situtations where a server on a 1 Gb/s link would cause the "NoResourcesPktsDropped" metric to increase on a 100 Mb/s access port when a client would download a large file, the metric would also increase more rapidly at the start of the transfer after which it would quickly stabilize (TCP's algorithms doing its magic I guess). This can be explained by the sender, at the beginning of the transfer, spitting out packets at over 100 Mb/s which would overrun the buffer (of the 100 Mb/s accessport or the 1 Gb/s port?) because the accessport ofcourse cannot keep up at > 100 Mb/s rates. I was however rather unaware that this could also pose a problem on devices that were connected at the same speed.

This might be a nice subject for a blog item as I've seen these metrics (NoResourcesPktsDropped, Discards, ...) confuse a lot of people (including me).

[qazzie]

I've also investigated the "NoResourcesPktsDropped" a bit and have seen situtations where a server on a 1 Gb/s link would cause the "NoResourcesPktsDropped" metric to increase on a 100 Mb/s access port when a client would download a large file, the metric would also increase more rapidly at the start of the transfer after which it would quickly stabilize (TCP's algorithms doing its magic I guess). This can be explained by the sender, at the beginning of the transfer, spitting out packets at over 100 Mb/s which would overrun the buffer (of the 100 Mb/s accessport or the 1 Gb/s port?) because the accessport ofcourse cannot keep up at > 100 Mb/s rates. I was however rather unaware that this could also pose a problem on devices that were connected at the same speed.

This might be a nice subject for a blog item as I've seen these metrics (NoResourcesPktsDropped, Discards, ...) confuse a lot of people (including me).

This is indeed an interesting point. The example you gave shows the way how frames are handled, this issue you suffered is known as micro-bursts... Try this on different switches and you really see different outcomes. The ers4500 isn't the strongest in line of switches when it comes down to this. And yes even 100mbit poort can increase the droponnoresources (without a loop offc.)

Maybe it's hard to make a blogpost about this, since there is almost no documention about it.

Q

[Dominik]

The buffer behavior is handled at a very low level inside the silicon, so there will be not much you can change on this behavior.

You will mostly deal with the buffers when many Ports are sending to one destination.
For example you have many gigibit ports that are sending traffic to one destination port, the port can not receive all that traffic,
here starts the buffer mechanism to cache the packets that can not deliverd. In fact that the buffer is limited in its capacity
the switch will start dropping on no ressources at this point when to much traffic is received and the buffer is overloaded.

All Network vendors have problems with that. On Cisco 6500 it depends on the line card sries how the buffer handles the overload.

 

[bylie]

...
Maybe it's hard to make a blogpost about this, since there is almost no documention about it.
...

Maybe it should be better documented than (hint Avaya , TCG...) as it could be an important metric in certain environments like iSCSI SAN's for example.

[slightly OT]
You know how these things can go in a larger company where each IT subdivision (netadmins, sysadmins, sanadmins, esxadmins, ...) has it's own people. A lot of the times, at first, nobody's equipment is to blame when something is acting strange/slow/... which can make it easy to point the finger at things which are not completely understood. Like "this metric on those switches is not 0 so it must the switches not cutting it, ofcourse it's not our storageboxes" while the underlying problem might be something totally else .
[/slightly OT]

At the moment we have an iSCSI SAN running on a stack of two 24 ports ERS 5510's where, on advise of our networkvendor, I've made the following adjustments to the switchconfiguration:

Code: [Select]

(config)# qos agent buffer maximum
(config)# qos agent queue-set 1
(config)# interface fastEthernet ALL
(config-if)# flowcontrol auto
(config-if)# exit
If I understand this correctly this will make the most use of the available bufferspace in the given environment (no QoS, no queues, ... is needed in a SAN). But what I'm wondering now is what could be the consequence to the traffic of the other ports in the portgroup when 100% of the buffer of that portgroup would be occupied by a port? Could it somehow also affect the traffic on the other ports in the portgroup?