Strange ARP table issue - 5520 and 8300

[DaveTheRave]

Hi all,

Stack of three 5520's, fully licensed for routing, Software version 6.2.1.. 8300 running latest code

I have a strange issue where devices are dropping out of the arp table on my switches. I have tried connecting voice equipment from three different manufactures, to three different vlans on these switches. Initially, when the equipment is first plugged in, I can ping the device from a different subnet and see its mac and ip entry in the arp table. After 5 to 7 minutes, I cannot ping the device and the arp entry has gone. If I then telnet directly onto the stack and ping the device, the arp entry comes back and the world is good again for 5 to 7 minutes.

The ARPLifeTime is set to 360 minutes and this is a very small network, maximum of 100 entries at any one time.

This is causing me an issue with a CS1000 upgrade to 7.5 as the DSP resources keep dropping off the network so registration is never complete.

I can get around this by adding static entries but really need to know why these devices are dropping out of the ARP table.

[Flintstone]

Hi DaveTheRave,

It sounds like that the actual 'ARPLifeTime' is set to 360 seconds rather than 360 minutes?

You might need a Sniffer, but have you confirmed whether your stack ARPs out for the devices when you ping from a different subnet?

CheerZ and good luck

[DaveTheRave]

Hi Flintstone,

Unfortunately, the sites I am experiencing this problem on are all remote with no local support so getting a packet trace is hard. It may take me sometime to obtain one.

Some further detail. I am on a totally different site and subnet to the stack and kit question. I can open a ping to the device, see it ping successfully for 5 to 7 mins then get no reply. I can only get the entry back into the arp table by pinging the device from something else on its local subnet. A reboot of the device also brings it back in.

[DaveTheRave]

Can you tell me how an arp entry gets created as dynamic or local?

The entries that are disappearing are dynamic.. Shouldn't they be local?

[Flintstone]

Hi DaveTheRave,

I believe 'local' means the IP addresses on the local interfaces.  Anything external will then be 'dynamic'?

CheerZ

[Michael McNamara]

A little background here with regard to the ARP timer and the MAC/FDB timer. I'm not sure if this translates directly from the ERS 8600 to the ERS 5000 so keep that in mind. If the MAC/FDB entry (default TTL is 5 minutes) expires the ERS 8600 will automatically remove/expire the any associated ARP entries. So the MAC/FDB aging timer needs to be set to 21601 (6 hours and 1 second) so that it will be longer than the default ARP timer (21601 seconds ~ 6 hours).

The question here is why isn't the switch/router ARPing for the device, or if it is why isn't the end device responding?

Software release 6.2.3 is available for the ERS 5000, you might want to try an upgrade to see if that provides a quick fix.

Which switch is performing the routing? The ERS 5000 or the ERS 8300? I'm guessing the ERS 5000 since you make mention of routing.

Cheers!

[DaveTheRave]

Well what do you know! All is working now. Despite this, we have our maintenance provider put a tap into our switch at each end of WAN and they are sniffing traffic to see if they can see anything unusual....

To answer your question though Michael, please see that attached diagram. I am in site B, trying to reach devices in sites D, C and A

[Michael McNamara]

That's a very interesting topology. I'm guessing that it's all Layer 3 (no Layer 2 between the sites?).

Are you using VRRP on the ERS 5520s or just a straight IP (Layer 3) interface on the specific VLANs?

Good Luck!

[DaveTheRave]

Is that interesting good or interesting bad? 

The WAN links are purely Layer 2, no QOS. It was cheaper for us to obtain huge amounts of bandwidth rather than use services like MPLS with QOS. We use queuing in the Nortel switches to ensure EF and other traffic is prioritised.

On the original problem, the sniffer has not shown any issues. Very odd.

[Michael McNamara]

If you are bridging VLANs (Layer 2) between the sites then you'd have a loop right? You'd need STP/RSTP/MSTP to block some of your redundant links. Are the WAN links just routed (Layer 3) links or are you using them as 802.1q trunk ports to bridge VLANs between the sites?

This is causing me an issue with a CS1000 upgrade to 7.5 as the DSP resources keep dropping off the network so registration is never complete.

What exactly is happening? You could have a QoS issue with a watchdog timer expiring and rebooting the IP phones. I'm guessing the IP phones register to the local CS1000 at the facility (across LAN) and not across the WAN, or are you running them in a survivable branch office mode where they register with a main office CS1000 and only use the local SRG/CS1000B as a PSTN gateway?

I've stayed with 6.1.5 software on all my ERS 5500 series switches... even the 5530s which are running OSPF because there have been too many issues with 6.2.x software for my liking.

Good Luck!

If this is just a simple IP (Layer 3) routed network I don't see where you would have any significant difficulties.

[DaveTheRave]

Your right Michael, we shouldn't have any difficulties but we have seen this issue once before. Having said this, the problem has cleared for now!

We are using OSPF to route traffic between sites which should take care of any loops.

We have 55xx 6.2.1 to 6.2.3, 8600 5.1.3.1 to 5.1.6 and 8300 4.2.3.0 to 4.2.3.3 upgrades planned.

Thanks for looking and I will update this post if we find anything.

[NortelNev]

apologies for bumping this thread but we are having a similiar issue with connecting to a few printers,switches (management interface only, normal traffic is fine) and also server's.

All connectivity stops except if you are on the same vlan as the device. Randomly thorughout the day these devices will drop in and out of periods of connectivity. I've narrowed it down to the device not being listed on the core switch's ARP table. We are able to restore connecivity to these devices if we telnet/ssh/etc into the device from a device on the same subnet and initiate outgoing traffic, eg ping. This then add's an ARP entry back into the core switch for the device and connecitvity is restored. Throughout the day though the ARP entry will disappear (even though all the devices are still on).

We are running a stack of Nortel ERS 5520's running 6.1.4.010 release as the core l3 switch. The issue is happening randomly on HP laserjet printers, Cisco 2950's (management interface mac address drop's out of core's arp table), and windows 2003/2008 servers on VMware ESX.

the ARPLifeTime is set to 360.

Hopefully someone has run into this issue and could shed some light!

The other alternative is we could set static arp entries on the core for the ~30 devices on our network this is happening to.

[NortelNev]

Just an update, have come across this update log, newbielink:http://blog.michaelfmcnamara.com/2010/08/ethernet-routing-switch-5000-software-release-v6-2-0-200/ [nonactive] , it appears this second bug fix might be related to issues we are having.

[Michael McNamara]

apologies for bumping this thread but we are having a similiar issue with connecting to a few printers,switches (management interface only, normal traffic is fine) and also server's.

All connectivity stops except if you are on the same vlan as the device. Randomly thorughout the day these devices will drop in and out of periods of connectivity. I've narrowed it down to the device not being listed on the core switch's ARP table. We are able to restore connecivity to these devices if we telnet/ssh/etc into the device from a device on the same subnet and initiate outgoing traffic, eg ping. This then add's an ARP entry back into the core switch for the device and connecitvity is restored. Throughout the day though the ARP entry will disappear (even though all the devices are still on).

We are running a stack of Nortel ERS 5520's running 6.1.4.010 release as the core l3 switch. The issue is happening randomly on HP laserjet printers, Cisco 2950's (management interface mac address drop's out of core's arp table), and windows 2003/2008 servers on VMware ESX.

the ARPLifeTime is set to 360.

Hopefully someone has run into this issue and could shed some light!

The other alternative is we could set static arp entries on the core for the ~30 devices on our network this is happening to.

While the symptoms might be the same your design is really completely different.

I'm guessing your not using IST/SMLT to connect your edge switches?

How long as the switch been up? I would probably advise an upgrade to at least 6.1.5 software to start with, if you haven't rebooted the switch in a while that might be a quick fix if you've been making a lot of changes. When you have these issues do you see the MAC address of the devices you are having issues communicating with anywhere in the MAC/FDB table?

I have note seen any issues like this when just running with a single ERS 5520 performing some routing.

Good Luck!

[NortelNev]

thanks for the reply, the 5520 stack has been up 250 days. I've put in a request to schedule an after hours restart.

the mac addresses of the device's don't appear in the mac address table either when this happens, in the meantime I have changed the mac-address-table aging-time to 1 week from the default of 5 mins. All of our DHCP leases are a week so hopefully this won't cause too many extra issues?

Cheers