8600 jumbo frame issues on older cards

[pgoggins]

We recently upped the MTU size to 9600 on our 8600 (8610 with a pair of 8692's) running 5.0.0.1 and have noticed when running the "show port info interface" command and all of the cards: 8608SX, 8608SXE,8630GB, and 8606GBE all show 9600 for the MTU size but the couple of 8616SX cards are still reporting 1950. I'm assuming since the 8608SX's are running at 9600 that jumbo frames are supported on non-E cards. Any ideas?

[Michael McNamara]

Hi pgoggins,

I believe that's just a display issue and I recall reading about it somewhere in some release notes.

You could test really quick using 'ping -f -l 2500' to a device that is downstream of the 8616SXE and supports Jumbo frames. If the packet doesn't pass through then you known you've got an issue with Jumbo frames not passing through the 8616SXE.

Important! - the device you are pinging needs to be configured to support Jumbo frames this generally requires that you edit the server NIC properties and enable 'Jumbo Frame' support.

Good Luck!

[pgoggins]

tried a couple of vmkping's with the MTU set between 5000 and 9000; all of the pings correctly worked through the 8616SX, thanks.

[tbigby]

In the release notes for version 5.1, it raises the following limitation (pg. 102):

Q01450691 - 8616GT/GTE modules discard jumbo frames by hardware design. The 8616GTE or 8616SXE modules do not support jumbo frames.

I saw this when upgrading to 5.1.1.1 earlier this year. Has anyone tried the 8616GTE modules with jumbo frames? We have 2x 8616GTE and 2x 8616SXE, and enabling jumbo frames would be good if possible.

Thanks!

[Michael McNamara]

It seems the information we have is at odds... the one thing I've learned from Nortel and other vendors is how carefully they word their documents. Just because something isn't "supported" doesn't mean it won't work or work properly. With that said it seems we need some additional clarification.

Is there anyone else out there that can perform a quick test to see if the 8616SXE or 8616GTE will pass Jumbo Frames through the module? You need to make sure that the "don't fragment" flag is set on the ICMP packet or the switches will just fragment the packet into 1500 byte sized chucks and then reassemble it.

Is there a "don't fragment" flag for vmkping?

We'll need to see if anyone else out there cares to help unravel the mystery here. I suspect that Jumbo Frames work but that the configuration is not supported due to the oversubscription (2:1) of the backplane and most notably the memory for buffering any large number of frames.

Anyone care to help us out?

[pgoggins]

There isn't a DF option for vmkping. I checked and the variant of an Intel Pro 1000 I was testing with didn't support Jumbo Frames, neither did the onboard BCM5721 chipset. I'm installing a BCM5708 (Broadcom NetXtreme II) which does support jumbo frames this morning and will be testing within a Server 08 guest os later today.

[pgoggins]

Ok, finished the tests.

Within ESXi 4 the following command needed to be run to enable jumbo frames (esxcfg-vswitch -m 9000 vSwitch0), the VM is a Server 2008 R2 Standard Edition install utilizing the VMXNet3 driver which has the MTU set to 9k for jumbo frame support. The physical network card is a BCM5807 PCI-E based card connected to a 4548GT-PWR with current firmware release (can't disclose anything more specific due to Avaya NDA) connected over gig fiber to an 8616SX (HW Rev 11) on slot 4 of a 8010 chassis running the 5.0.0.1 release with the MTU size set to 9600.


I am able to use "Jumbo Frames" to a limited extent. I can run pings up to 1892 which would mean a maximum MTU size of 1920. I also tried moving to an 8630GBR but was seeing the same results. After setting the MTU to 9600 does the 8600 need to be restarted?

[Michael McNamara]

It's my understanding that the switch needs to be restarted if you change the MTU, yes.

Also I don't believe the ERS 4500 series switches support Jumbo Frames. I know that in 6.1 software for the ERS 5000 series switches they support Jumbo Frames. You can enable Jumbo Frames with the following command on the ERS 5000 series switches that are running 6.1 or later;


jumbo-frames enable

[pgoggins]

The 4500's have jumbo frames enabled by default. I did some followup testing from within the same vlan and I am able to pass 9000k packets fine on the 8630GBR. I'll do some testing later today after migrating back to the 8616SX. This does show that across vlan's frames over 1920 are not being passed correctly (at least without a reboot).

[Michael McNamara]

I updated my previous post after you posted here...

"I don't believe the ERS 4500 series switches support Jumbo Frames. I know that in 6.1 software for the ERS 5000 series switches they support Jumbo Frames."

A little research or a few phones call might be required to confirm or deny that theory.

[pgoggins]

The 4500's have native support for jumbo frames along with the 2500's (though it's pretty useless on the 2500's) while on the 5000 and 8000 series it is supported by newer firmware releases but is disabled by default.

Summary of test results:
8616SX - limited jumbo frame support to 1920
8608SXE - works up to 9000k
8630GBR - works up to 9000k

[Michael McNamara]

Thanks for taking the time to verify, +1 for pgoggins.

I just found it curious that there was no 'option' within either Device Manager or the CLI interface to configure the MTU for the Avaya/Nortel Ethernet Routing Switch 4500 series. I only have access to a single ERS 4548 running 5.1.2.x software so perhaps there are additional options available in 5.1.3.x software?

What if I intentionally wanted to set the MTU to 1500 on the ERS 4500?

Anyway, I would guess then that the 8608GTE would perform similarly to the 8608SXE. And I would also guess that the 8616GTE performs similarly to the 8616SXE.

Cheers!

[tbigby]

Thanks for the info pgoggins, very helpful. We'll leave jumbo frames disabled then.

I can't see any way to set the MTU to 1500 on the 4500s, although we are running an older firmware (5.1.1). Not that I want to though!

[pgoggins]

The 4500's have no option to disable jumbo frame support. One thing to remember even if Jumbo Frames are enabled on your network systems will not use jumbo frames unless it is explicitly supported by the network card and you enable the feature.