Broadcast and multicast rate limit values - best practice

[Michael McNamara]

Here's another topic that I get a lot of questions on. What's the best practice values for configuring broadcast and multicast rate limiting in "my" network.

Here's what I've used to date (which has been tweaked in the past);

• Ethernet Switch 460/470 - All ports rate limit broadcast and multicast at 5%
• Ethernet Routing Switch 4500/5500/5600 - All ports rate limit broadcast and multicast at 5%
• Ethernet Routing Switch 8600 - Use CP-Limit values of 3000 (pps) for multicast and 2500 (pps) for broadcast
• Ethernet Routing Switch 8600 - All ports rate limit broadcast and multicast at 5000 (pps)
  

In reality I've seen devices on a network start to 'freak' out at just around 300pps (packets per second) in an ARP storm or broadcast storm.

I should warn you that you don't want to set the rate limiting value lower than the CP-Limit value on the Ethernet Routing Switch 8600 or the CP-Limit threshold may never kick-in. I've have CP-Limit and SLPP (and VLACP) save the day on numerous occasions.

Those are the values that I use today in my networks. If you utilize a lot of multicast traffic such as video/audio streaming then you may need to tweak those values accordingly.

What values do you use?

Cheers!

[edslopes]

Hi Michael,

Here´s what i´ve used for our major customer:

    * Ethernet Switch 460/470/4500 - All ports rate limit broadcast and multicast at 2%
    * Ethernet Routing Switch 8600 - Use CP-Limit values of 5000 (pps) for multicast and 5000 (pps) for broadcast for Server Access. Extend CP Limit = SoftDown
    * Ethernet Routing Switch 8600 - Use CP-Limit values of 2500 (pps) for multicast and 2500 (pps) for broadcast for Uplink Access(SMLT). Extend CP Limit = SoftDown
    * Ethernet Routing Switch 8600 - All ports no rate limiting broadcast

CP Limit feature saved the day 6 months ago.

Regards,

Edson

[Michael McNamara]

Thanks Edson!

Is it possible that you have your Server access and Uplink access numbers transposed?

I would think you would need larger numbers on your MLT/SMLT uplinks to account for multiple VLANs, etc than the individual servers themselves.

Thanks for sharing your information!

[qazzie]

What do you guys use for uplinks rate limiting on access stacks/switches?

2% on access ports and 5-10% on access uplinks?

And the downside of rate limiting is the notify, there is no snmp-trap,syslog-msg or dram-log-msg saying that rate-limiting is dropping traffic due to the threshold. For now can use rmon traps but anyone using anything smart?

Ow and worth mentioning, when using (ext)-cp-limit it only works if you've an active ERS ip-interface in the vlan traversing the port.

Besides the mentioned hardening features no loop-detect, use SLPP, use VLACP, use BPDU-filtering on access as well as STP-FastStart.

Leaving cp-limit to default values 10k/pps, using cp-limit on the ports. Only when having a multicast ingress stream adjusting cp-limit and taking ext-cp-limit of the port
 
regards
q

[Michael McNamara]

2% on access ports and 5-10% on access uplinks?

I think those values would be fine, for ease of use and configuration I just use 5% on all ports.

And the downside of rate limiting is the notify, there is no snmp-trap,syslog-msg or dram-log-msg saying that rate-limiting is dropping traffic due to the threshold. For now can use rmon traps but anyone using anything smart?

This is very true. The rate-limiting is performed in hardware on the ASIC which is really the only way is survives the onslaught of all that traffic. If you are monitoring ifInOctets and ifOutOctets on your uplinks/downlinks you will still see a very signifcant spike in utilization and then generate an alarm off that threshold failure.

Besides the mentioned hardening features no loop-detect, use SLPP, use VLACP, use BPDU-filtering on access as well as STP-FastStart.

All very highly recommended...

Leaving cp-limit to default values 10k/pps, using cp-limit on the ports. Only when having a multicast ingress stream adjusting cp-limit and taking ext-cp-limit of the port

I would recommend that you NOT use the the default CP-Limit values of 10,000. In my testing the values are too large to protect the network in a timely manner. In some cases the 8691SF CPUs became overwhelmed with the traffic before CP-Limit could kick in leaving the network dead in the water.

Cheers!

 

[qazzie]

Hi micheal,

Not really posting to defend myself. But I saw I made a little typo, which offcourse can be very dramatic in our playfield

"The CP Limit function protects the CPU by shutting down ports sending Multicast or Broadcast traffic to the CPU at a rate greater than desired through one or more
ports. The Extended CP Limit functionality is configurable and can be used to protect the switch from being overwhelmed by any kind of traffic."

So all ports hardenend by ext-cp-limit preferred, and if that can't be done due to certain traffic patterns I take ext-cp-limit off the port and falling back on traditional cp-limit with counters in the range between 3000-6000. And indeed not letting it default. On ports where ext-cp-limit is configured I let cp-limit as it -default- because it's protected by ext-cp-limit.

cheers m8
q

[Michael McNamara]

This has been a great discussion with your input Qazzie, always interested in reading more.

You are very correct, here's the exact text from Nortel (The Large Campus Technical Solution Guide)

The ERS 8600 supports the ext-cp-limit feature which goes one step further than cp-limit by adding the ability to read buffer congestion at the CPU as well as port level congestion on the I/O modules. This feature will protect the CPU from any traffic hitting the CPU by shutting down the ports which are responsible for sending traffic to CPU at a rate greater than desired.

There might be a blog post here somewhere...

Thanks as always for participating in the discussion!