• November 29, 2020, 06:24:54 PM
Welcome, Guest. Please login or register. Registration is free.
Did you miss your activation email?

Author Topic: ERS8600 v7.2.0.2 Cluster - Mac Entries Reached 64000 Error  (Read 3503 times)

0 Members and 1 Guest are viewing this topic.

Offline tammamih

  • Rookie
  • **
  • Posts: 4
ERS8600 v7.2.0.2 Cluster - Mac Entries Reached 64000 Error
« on: November 28, 2013, 08:03:51 AM »
Hi,

We are facing an issue in a simple IST/SMLT cluster consisting of 2 ERS8600 cores  running v. 7.2.0.2 and ERS5520 edge switches running SW6.2.4 & other ERS5520 edges running SW5.1.4. The issue appears constantly (once a week) and sometimes once a month.

Description of issue:
Network Slowness and High CPU in Core Switches ERS8600. During the issue, the following error log messages appear:
"CPU5 [11/26/13 15:24:05] SW ERROR Number of Mac Entries Reached 64000\0x0AMac Address learning stopped in r-mode"

After investigation we found the issue by running the following command:
>>show vlan info fdb-entry
we found the one vlan ID is having over 30,000 "Fake" unknown mac address learned in it
So we go to the edge switch where the vlan is configured:
we found that one access port connected to one PC "Windows 7". This port has many unknown "30,000" mac address learned in it, very strange!!
When we disable this port, network slowness disappear and CPU utilization drops and everything is back working normally.
After one hour we enable the port and it correctly learns only 1 mac-address "windows 7" PC. Windows 7 has been virus scanned and it's found clean.

We have physically inspected this port connection and it is not connected to a hub or a switch. This port got BPDU and spanning tree enabled.

The same issue happens after a while in different port in a different edge switch. We encounter this problem from time to time "once a week, once a month" there is not really a timeframe where this problem occurs.

I appreciate your support .


Offline TankII

  • Hero Member
  • *****
  • Posts: 556
Re: ERS8600 v7.2.0.2 Cluster - Mac Entries Reached 64000 Error
« Reply #1 on: November 29, 2013, 08:21:37 AM »
Are you sure that PC's user isn't going out to reverse proxy sites and to get around corporate rules?  We had a similar issue and it turns out a doctor was using an SSL tunneling site to reverse proxy out and watch on-line Videos.  Since it's a tunnel, you could have connections try and come backwards into your network, which could give you lots of invalid MAC entries since the PC isn't capable of processing them directly.
If you have OSPF active on your user network interfaces, you could also have a route injection if you are not careful with a scenario like this.

TankII

Offline tammamih

  • Rookie
  • **
  • Posts: 4
Re: ERS8600 v7.2.0.2 Cluster - Mac Entries Reached 64000 Error
« Reply #2 on: November 29, 2013, 09:05:38 AM »
Yes, all internet access are protected/monitored by Bluecoat Proxy SG appliance, where it blocks "proxy avoidance" sites and so on. Is there a way to completely block this functionality from Firewall or Group Policy ?? is it a PC issue or Network issue "Switch/Routing" ?

Best Regards,

Offline tammamih

  • Rookie
  • **
  • Posts: 4
Re: ERS8600 v7.2.0.2 Cluster - Mac Entries Reached 64000 Error
« Reply #3 on: December 04, 2013, 08:10:56 AM »
To keep you updated, below summarizes our findings:

We have noticed this problem occurs on few machines only when it is in hibrantate/Sleep mode state! So this exclude that fact that it is related to web access. Would you think it is related to bugs within Windows PC network cards?

This is one thing i found in common between those machines, they are all using same network card: Intel 82579LM Gigabit Ethernet. Keep in mind that we have over 400 machines using this network card, but problems happend randomally, where one PC keep giving those random mac address.

Also, do you suggest enableing Mac Security to permit switch ports of autolearning only 2 mac addresses? would that be workaround to this issue?

I appreciate your thoughts and feedback.

Offline TankII

  • Hero Member
  • *****
  • Posts: 556
Re: ERS8600 v7.2.0.2 Cluster - Mac Entries Reached 64000 Error
« Reply #4 on: December 05, 2013, 01:59:13 PM »
Yes, you can try setting up MAC security - it's good practice anyway.
Odd it's only on these computers.  Sounds like a bootROM flash is required or a BIOS update as only they talk hardware for WOL when the PC is down.

TankII

Offline tammamih

  • Rookie
  • **
  • Posts: 4
Re: ERS8600 v7.2.0.2 Cluster - Mac Entries Reached 64000 Error
« Reply #5 on: December 10, 2013, 01:58:54 AM »
Update:

We have disabled Hibrernate/Standby through GPO to all Windows Machine to confirm that the issue is definitly caused by Standby Mode. It has been 7 days and the problem seems gone so far.

We will monitor it for a month. If all go well, we will confirm cause of issue and update all NIC drivers as a permanent solution.

Regards,