• September 25, 2020, 02:28:54 AM
Welcome, Guest. Please login or register. Registration is free.
Did you miss your activation email?

Author Topic: QoS/Buffer question ERS56xx  (Read 4529 times)

0 Members and 1 Guest are viewing this topic.

Offline ijdod

  • Rookie
  • **
  • Posts: 20
QoS/Buffer question ERS56xx
« on: October 07, 2013, 03:28:31 AM »
I've noticed something going over our ERS5650 config. This config is factory default as far as QoS in concerned.

By default, this switch is configured with two queues (queue-set 2). Buffers are distributed between the two queues (roughly 3/5 queue 1, 2/5 queue 2). All ports are untrusted, no classifiers are assigned.

To me this would imply that we're only using about 2/5 of the available buffer space for *all* our traffic. All traffic is assigned to the lowest priority due to the untrusted nature of the interfaces, and lack of classifiers. This means assignment to queue 2, with the associated smaller buffer allocation. Some tuning can be done with the general buffer strategy (regular/large/maximum), but the above would remain valid in principle.

Am I missing something here, or is my reasoning correct?

Following from this, I can see two ways of improving the use of the buffers:
1. rebooting to queue-set 1.
2. Classifying some traffic so it ends up in queue 1.

(we're troubleshooting very bad performance (as in: far worse than expected for this class of switch) under heavy congestion conditions. Throwing bandwidth at it would be the proper solution, but that's not in the cards unfortunately.  )



Offline TankII

  • Hero Member
  • *****
  • Posts: 556
Re: QoS/Buffer question ERS56xx
« Reply #1 on: October 07, 2013, 08:55:52 AM »
Actually...
There are eight queue's.  Queue 2 is set as the preferred queue, that's all.  Each queue has equal amount of starting memory in the default configuration of normal.
Normal traffic is in queue 1, untrusted, but since the buffer is 'normal' by default, no traffic should overrun any queue.

If you are running iSCSI, the recommended configuration is queue-set 1, buffer maximum (if Layer 3) buffer lossless if running L-2.  So, if you are not running your VOIP environment on these switches (requiring queue 8, buffer large) and want increased performance, change it to 1 and maximum.  You will not be able to run anything with QOS enabled on the switch as the dscp trusted ports will want to choose a queue other than 1 and their will be insufficient memory allocated for other classes of service.

TankII

Offline ijdod

  • Rookie
  • **
  • Posts: 20
Re: QoS/Buffer question ERS56xx
« Reply #2 on: October 08, 2013, 02:22:17 AM »
Actually...
There are eight queue's.

According to the documentation there's anything from 1 to 8 queues, depending on the actual queue-set you chose. Default is queue-set 2, for 2 queues.

Are you perhaps mistaking queues with priorities?

Quote
  Queue 2 is set as the preferred queue, that's all.

Preferred queue doesn't make sense to me in this context.

Quote
  Each queue has equal amount of starting memory in the default configuration of normal.

As far as I can tell from the documentation:

There's the buffer sharing strategy (standard, large, maximum), which configures how the total buffer memory is shared between 12 ports. Even standard does not divide this up equally (16% max per port, for 12 ports. ~8% would be an equal share). On each port, the 1 to 8 configured queues do not get an equal share of the memory available to the port. This is supported again by documentation, and the out of of the 'show qos queue-set' command. 

Quote
Normal traffic is in queue 1, untrusted, but since the buffer is 'normal' by default, no traffic should overrun any queue.

The default Untrusted policy assigns priority 0. This ends up in queue 2.

Queue Set 2
802.1p Priority Queue
_______________ _____
0               2
1               2
2               2
3               2
4               2
5               2
6               1
7               1


Traffic will overrun a queue if there's (enough) congestion.

Quote
If you are running iSCSI, the recommended configuration is queue-set 1, buffer maximum (if Layer 3) buffer lossless if running L-2.

This would confirm my suspicions, as this would indeed mean you configure all of the buffers to a single queue, and assign the maximum possible depth to each buffer (at a trade-off elsewhere).

Offline Michael McNamara

  • Administrator
  • Hero Member
  • *****
  • Posts: 3842
    • michaelfmcnamara
    • Michael McNamara
Re: QoS/Buffer question ERS56xx
« Reply #3 on: October 08, 2013, 11:49:17 AM »
Hi ijdod and welcome to the forums!

Your assessment is spot on, however, you need to consider why the switch is having to buffer anything.

If the switch is capable of wire speed routing and switching then you have an overloaded server or network link somewhere else that is causing the problems. I'll often see this problem in an overloaded server which can't keep up with the traffic from the clients.

The problem will also depend on which protocol you are utilizing, while there are mechanisms built into TCP to accommodate congestion and throttling (TCP Window Size), there's nothing in UDP to help pace the traffic flow.

Have you looked at a packet capture to see what's really going on?

Good Luck!
We've been helping network engineers, system administrators and technology professionals since June 2009.
If you've found this site useful or helpful, please help me spread the word. Link to us in your blog or homepage - Thanks!

Offline ijdod

  • Rookie
  • **
  • Posts: 20
Re: QoS/Buffer question ERS56xx
« Reply #4 on: October 09, 2013, 02:59:01 AM »
Hi ijdod and welcome to the forums!

Your assessment is spot on, however, you need to consider why the switch is having to buffer anything.

If the switch is capable of wire speed routing and switching then you have an overloaded server or network link somewhere else that is causing the problems. I'll often see this problem in an overloaded server which can't keep up with the traffic from the clients.

The problem will also depend on which protocol you are utilizing, while there are mechanisms built into TCP to accommodate congestion and throttling (TCP Window Size), there's nothing in UDP to help pace the traffic flow.

Have you looked at a packet capture to see what's really going on?

Good Luck!

Thanks. Fairly new to the Nortel/Avaya gear, from a Cisco/HP background. The ride has been... interesting so far :D.

I agree the problem is the actual traffic, and optimising the buffer use will mostly be drops in the ocean. It's a fairly classic congestion issue. Because of reasons we ended up with a couple of 5650 switches, each with a 4xGbE uplink. Each is nearly fully populated, mainly with ESX servers. Adding insult to injury, the virtual hosts are backed up over the LAN. We're in the process of solving this the proper way, but we're looking into any quick wins along the way as well.

Edit: Our VAR sings a different story, claiming the queues dynamically allocate the memory. Until convinced otherwise, I'm going to assume they're confusing the per port allocation (large, maximum, and so on) with the queue-set allocation within that allotment.
« Last Edit: October 09, 2013, 09:21:13 AM by ijdod »

Offline TankII

  • Hero Member
  • *****
  • Posts: 556
Re: QoS/Buffer question ERS56xx
« Reply #5 on: October 09, 2013, 11:34:22 AM »
I'll give a little background that may explain my explanation vs. current documentation.
When the 5510 first came out, each of the eight queues was fixed in size.  The default config of 2, normal caused extreme havoc with servers that were at 100Mbit alongside others that were at GIG.  If all servers were at the same speed in the switch, there were no buffer issues.  With a mix-n-match, things were godawfull slow for some servers.
We switched to 1 and large, which allowed the majority of the memory to be hard-allocated to queue 1, the default queue for the switch for untrusted traffic (DSCP=0), with the rest at significantly smaller allocations.  This was the explanation we were given by Nortel Product Development in '05.  You could pick and choose the queue you wanted to have the priority for memory allocation to with the number of the queue-set.  We were not running VOIP at the time, and not within the Server Farm, so this was not a problem.

If they changed the architecture since 5.1 came out and updated the documentation to go with it, then it's my fault for not keeping a closer eye to the changes.

TankII

P.S.  The queues being 1-8 donít match DSCP 0-7 by number, but thatís how the queue relationship was explained to us many, many moons ago.

Offline bylie

  • Sr. Member
  • ****
  • Posts: 149
Re: QoS/Buffer question ERS56xx
« Reply #6 on: October 10, 2013, 03:25:11 AM »
Maybe the attached Avaya document might shed some light on this, especially p16-20. It also explicitly states that prior to SW v4.0 there was only a single 8 queue queue-set which might be what TankII was referring to. Keep in mind this document refers to the ERS 5500 series, so it might not be applicable to the ERS 5600 series as I'd expect those to have different (as in higher) specs.
The way I always understood this queueing thing was that with the queue-set assignment you select a certain fixed queue configuration which, together with the global buffer setting, determines how many and how large (deep) the queues are going to be. The 802.1p/CoS queue mapping table, after first doing DSCP to CoS mapping, then does the final job of assigning egress packets to the correct queue. Then, and only if an interface is actually congested, are the different queues and their queueing disciplines used to effectively prioritize certain packets thus giving them better service.
Regarding the mixing of 100 Mb/s and 1 Gb/s ports, might this also not have been a flow-control issue in which the serverswitch might have been sending flow-control frames upstream because a 100 Mb/s server port was congested? I remember reading about flow-control and how it could wreak havoc when inproperly configured.

Offline ijdod

  • Rookie
  • **
  • Posts: 20
Re: QoS/Buffer question ERS56xx
« Reply #7 on: October 10, 2013, 03:31:23 AM »
TankII: Thanks, your reply makes a lot more sense now, seems like we're talking about the same thing. From the docs I gather than Avaya changed to their current setup (from fixed 8 to configurable 1-8) with release 4. Well before my time, we're mostly on 6, with some 5 thrown in to keep it interesting.

bylie: Thanks, that was one of the documents I found, and also one of the few Avaya docs that actually seem to go indepth. A lot of other Avaya docs don't even mention what the large/maximum settings actually do...