DHCP snooping issue ERS4850 5.8.2

[Jeroen]

Just to inform those who are using ERS4850 with 5.8.2 code. I just had this code installed on several ERS4850 stacks last weekend and this week strange connectivity issues started happening.
After some troubleshooting, I noticed that several connected devices (access points, clients, audio/video devices) were not receiving a valid IP-address.
Since DHCP server was working correctly, as well as some other devices and clients still were communicating on the same switch stack, I did set the DHCP snooping setting on a specific interface to trusted. And voila, the device immediately started working again!
This issue is currently under investigation by Avaya and seems to be a DHCP snooping bug in 5.8.2.
As I workaround of this issue set all ports to trusted or disable DHCP snooping globally. Be aware that this can allow DHCP rogues to be actrive in the network.

Lets hope Avaya is able to resolve this issue quickly.

Jeroen

[Telair]

Nice find.  Thanks for letting us know!

[Jeroen]

Even the cause of the issue is still not known, I think it would be interesting to share the current status of the investigation with you.
So far, we have learned that the issue of clients not receiving DHCP is due to the DHCP snooping binding table filling up with entries, while the ERS4850 is not freeing the table ending up with hitting the entry limit of the binding table.
The root cause of this is still not known. Last week Avaya shared a specific image version in which debugging has been enabled regarding the DHCP snooping functionality.
Ive prepared a syslog server whih receives all log events of the ERS4850. During previous troubleshooting tests, the issue in my particular environment occurs after around 2 weeks. Will probably have to wait 10 more days for the connectivity issue to occur again. Hopefully the syslog shows the reason behind this issue.

To be continued

[telecom116]

Thanks for the update.  Keep us informed as this goes along.

[pat2012]

Hasn't this issue been well documented in Avaya's release notes since Rel. 5.6.4?

"A DHCP memory leak issue was addressed in this release that included a change in the DHCP packet header.

In code versions prior to 5.6.4, the code added 4 bytes to each egressing DHCP packet without changing the total length value of the packet thus creating a malformed DHCP packet. The 5.6.4 release will now discard these packets when DHCP snooping is enabled.

This fix may create unexpected loss of DHCP packets when the 4k is connected to other ERS switches running prior code. The affected ERS switches are 2500/3500, 4k running code prior to 5.6.4, and 5k running code prior to 6.3.3.

The workaround is to disable DHCP snooping until this fix is propagated to all ERS switches."

[Telair]

Software release v5.8.3 just came out today and it had some notes on DHCP issues.

- ERS454800-1448 - Clients were not getting IP address on few specific access ports

Maybe a code update will fix your issue now?

https://downloads.avaya.com/css/P8/documents/101020083

[Jeroen]

Hi Telair, thanks for the headsup.

I've been working with Avaya on this issue for several months now. I had a 5.8.2 debug image (private build) which should help Avaya getting the required information. Ever since, the issue did not occur anymore.

Meanwhile Avaya stated that this issue will be fixed in 5.9.3 which will be released at the end of May. Until that time, I will have the DHCP snooping feature disabled as clients need to be able to get connectivity.

Surely, I have to test this new code to be certain the issue has been fixed or not.

[Jeroen]

Hi all, it has been some time for me to update this thread, but eversnce I've been busy troubleshooting this issue with Avaya.
Unfortunatey we haven't been able to find the root cause of this issue.
The great news is is that with the new code 5.9.3 the issue is solved. Not sure what exactly fixed it but there has been several snooping improvements made.

Finally got DHCP snooping working again preventing rogue network adresses when the script kiddies setup their own DHCP service again.