Creating SMLT links disconnects clients from OTHER switches

[tfboy]

A bit of a strange one this. Coming from Cisco background, I'm trying to get my head around setting up some redundant MLT links.
I have a typical VSP9000 cluster linked by a twin 10Gb IST link. All L3 routing is done on the VSP and they are setup as VRRP.
I have an ERS5650 connected to one of the VSP9000 over 10Gb, working fine. Standard trunk connection with a bunch of VLANs tagged on the link.
Now, I want to connect a second 10Gb link to the other VSP, creating your typical triangle configuration. I have created an MLT with the same ID on both VSPs, and associated the correct port and set it to SMLT. When I create an MLT on the 5650 associating both ports, all looks OK. I can see I have an MLT of type SMLT in the VSP9000. If I ping the management address of the 5650 from a PC connected on the VSP, it works, I can unplug one fibre it works, plug it back in and unplug the other fibre, it still works, so all looks OK.

BUT, I have a second 5650 still single-attached to just one VSP, and as soon as I have the first 5650 running with dual connections, I lose all connectivity to this other 5650. As soon as I disable one of the two links on the second 5650, connection resumes on the first.

[Telair]

According to your diagram you are using the same MLT/SMLT entry (51) for both switches from your VSP 9000's?  If they are separate switches, they need separate MLT/SMLT entries.  If the switches were in a stack together, then they could use the same MLT/SMLT entry.  If it is how your diagram shows, the VSP 9000's are assuming the two switches are one stack or unit with 4x SMLT links.

[Dominik]

Year as mentioned by telair create a unique MLT / SMLT ID for each switch/stack you want to connect and it should solve your problem.

[tfboy]

Whoops, sorry, bad copy/paste in making the drawing. The VSPs have unique MLT numbers: 51 and 52. The 5650 each uses MLT #1 for the link. I've edited the original drawing.

The Avaya documentation says the MLT number is common across both switches.

Taking the Avaya design doc NN46250-503_02.02 (cannot post the link), it suggests on page 21 that the MLT number should be the same, see attached drawing.

There are only two differences in my case: the "edge" switch is a 5650, not another VSP9000 and I want 802.1q tagged trunking so several VLANs transit over the SMLT and not just VLAN 10.

[Dominik]

Do you also have configured a SMLT ID for these two MLTs ?

Would be great if you could provide the config of your VSP9k, so that I can check what is missing here.

[tfboy]

Hi Dominik,
I think I've found the problem: there was a normal GbE connection with same VLAN number as the SMLT which was used as a management connection for the 5650, so I guess I had created a loop and this shut down the 10GbE connection to this switch.

If only Avaya had a proper out of band network connection. The two ports on the back of 5650s are not used?

I need to do some more checks and will let you know.
I feel a little stupid not spotting this earlier. I'll post again with an update once I've had a chance to do some more testing.

[Dominik]

The out of band Port on the ERS5600 was introduced with SW release 6.6:

Out-of-band management
Out-of-band management allows switch or stack management through the dedicated out-ofband
management port. This port can accept an IPv4 or IPv6 address different from the switch
or stack IP address. With out-of-band management, you do not require an in-band
management VLAN to carry switch or stack management traffic, including Telnet, Secure Shell
(SSH) protocol, Simple Network Management Protocol (SNMP), HTTP, or HTTPS. You can
also use out-of-band management to download a software image or access the Enterprise
Device Manager (EDM) interface for a switch or stack.
For more information, see:
• Getting Started with Avaya Ethernet Routing Switch 5000 Series, NN47200–303

If you are using In-band managemnt I would recommand to put the management VLAN on the normal SMLT/MLT Uplinks instead of pacthing an extra cable. An extra cable for management is always a risk
as you have seen in your case.

[tfboy]

Thanks. The switches are currently on 6.2 so will look at getting the more recent firmware and update them and consider migrating the management IP over to those interfaces.

In the meantime, we can either include the management VLAN in the MLT and not have a dedicated connection for it. It would be safer, but of course any issues with the MLT and we've cut ourselves off, so I'd prefer the out of band connection which is totally seperate.

[tfboy]

OK, it turns out it wasn't my extra connection causing the problem: it seems to be related to the fact that both seperate 5650 switches were using MLT 1 which it didn't like. As soon as I plugged in a second, it killed the uplinks to the first.
Now I've specified a different MLT number on the 5650, it seems ok (using 1 on the first, 2 on the second).
However, I still appear to have some issues. If I pull the fibre to test failover, it works fine on the second 5650, but the first is a little slow: I have quite a few packets dropped when I restablish the 2nd link. I've quickly scanned over the configurations on both 5650 and 9000 and they're consistent across both MLTs so I must be missing something.
Also, after some time, there's a lot of dropped packets, then it will come back. It's as if traffic doesn't get through every now and then.

For the moment, I've disabled the second set of fibre links on both connections and it appears to have stabilised again.

I'll try to extract the relevant configs and post here.

[tfboy]

I had another look at this and found the cause of the problem: for some reason, the MLT link wasn't activating on one of the 10GbE connections, so there was a small loop and as I had activated rate limiting, the port was shut down after a while.

After removing and re-enabling the MLT configuration, it seems to stick and all interfaces remain up with no errors.

Not quite sure why it didn't work first time round, but all OK now.

Thanks for your tips. Now onto my new issue: since activating IPFIX, I can no longer extract configuration from CLI: a "show run" crashes my session!

[Dominik]

Thanks for the feedback, good to hear that you have ound the problem and everything is working now.

Cheers