• September 19, 2020, 02:32:21 PM
Welcome, Guest. Please login or register. Registration is free.
Did you miss your activation email?

Author Topic: *Resolved* ERS5698 - Added new MLT and VLANS were removed from other MLTs  (Read 4902 times)

0 Members and 1 Guest are viewing this topic.

Offline brazenhead

  • Sr. Member
  • ****
  • Posts: 132
I experienced an odd issue (that caused an outage) that I will try to explain and summarize as best as possible. If anybody has ever heard of anything like this or notices where I might have erred, please let me know so I that can ensure it doesn't happen again (plus I need to provide an explanation).

I have an ERS5698 IST clustered pair of switches running SW:v6.3.1.039, FW:6.0.0.16. The switches consist of 14 SMLTs spread across the 2 switches and an IST MLT link. The SMLT's all have the same VLANs configured on them.

Basically what happened is I created a new MLT, SMLT, added the same VLANs and for some reason those VLANs were automatically removed from all of the other existing SMLT's and access and trunk ports.

Here are is the MLT/SMLT/VLAN portion of the switch config:
Code: [Select]
vlan members 3000 1-24,47-48,93-96
vlan members 3001 1-24,47-48,93-96
vlan members 3002 1-24,47-48,93-96
vlan members 3999 47-48

mlt 1 enable member 1-2
mlt 2 enable member 3-4
mlt 3 enable member 5-6
mlt 4 enable member 7-8
mlt 5 enable member 9-10
mlt 6 enable member 11-12
mlt 7 enable member 13-14
mlt 8 enable member 15-16
mlt 9 enable member 17-18
mlt 10 enable member 19-20
mlt 11 enable member 21-22
mlt 12 enable member 23-24
mlt 29 enable member 93-94
mlt 30 enable member 95-96
! IST Link
mlt 31 enable member 47-48

interface FastEthernet ALL
spanning-tree port 1-24,47-48,93-96 learning disable
exit

mlt spanning-tree 1 stp 1 learning disable
mlt spanning-tree 2 stp 1 learning disable
mlt spanning-tree 3 stp 1 learning disable
mlt spanning-tree 4 stp 1 learning disable
mlt spanning-tree 5 stp 1 learning disable
mlt spanning-tree 6 stp 1 learning disable
mlt spanning-tree 7 stp 1 learning disable
mlt spanning-tree 8 stp 1 learning disable
mlt spanning-tree 9 stp 1 learning disable
mlt spanning-tree 10 stp 1 learning disable
mlt spanning-tree 11 stp 1 learning disable
mlt spanning-tree 12 stp 1 learning disable
mlt spanning-tree 29 stp 1 learning disable
mlt spanning-tree 30 stp 1 learning disable
mlt spanning-tree 31 stp 1 learning disable

interface mlt 31
ist peer-ip 10.1.1.2
ist vlan 3999
ist enable
exit
interface mlt 1
smlt 1
exit
interface mlt 2
smlt 2
exit
interface mlt 3
smlt 3
exit
interface mlt 4
smlt 4
exit
interface mlt 5
smlt 5
exit
interface mlt 6
smlt 6
exit
interface mlt 7
smlt 7
exit
interface mlt 8
smlt 8
exit
interface mlt 9
smlt 9
exit
interface mlt 10
smlt 10
exit
interface mlt 11
smlt 11
exit
interface mlt 12
smlt 12
exit
interface mlt 29
smlt 29
exit
interface mlt 30
smlt 30
exit

And once I applied the following changes to create the SMLT and add the VLANs to the unused ports 91, 92:

Code: [Select]
vlan ports 91,92 tagging tagAll filter-untagged-frame enable
vlan members 3000 91,92
vlan members 3001 91,92
vlan members 3002 91,92

interface FastEthernet ALL
spanning-tree port 91,92 learning disable
exit

mlt 28 name "new MLT" enable member 91,92
mlt spanning-tree 28 stp 1 learning disable

interface mlt 28
smlt 28
exit
save config

VLANs 3000, 3001, and 3002 were automatically removed from all of the other MLT's! Any idea what might have gone wrong? As expected, the Avaya ERS logs show nothing useful.Hopefully it is not my "unconventional" numbering of MLTs, SMLTs - I can't see that causing this type of issue.

If you think more details might help to clarify things, let me know and I will add as much as I can.

Thanks in advance.
« Last Edit: February 03, 2014, 07:51:38 AM by brazenhead »


Offline Paul L

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 754
    • paulaleroux
    • Paul's Networking blog
Re: ERS5698 - Added new MLT and VLANS were removed from other MLTs
« Reply #1 on: January 31, 2014, 09:19:44 PM »
did you by chance do this from the Web EDM interface?
ACSS- Avaya Enterprise Routing Switch  #8

Offline Paul L

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 754
    • paulaleroux
    • Paul's Networking blog
Re: ERS5698 - Added new MLT and VLANS were removed from other MLTs
« Reply #2 on: January 31, 2014, 09:21:44 PM »
un-related.
Its not a good idea to make your last MLT the IST.

It's Avaya's best practice to make the IST the MLT1.  There has been observed stability issues in switch clusters.hen the MLT 1 is not used as the IST.
ACSS- Avaya Enterprise Routing Switch  #8

Offline brazenhead

  • Sr. Member
  • ****
  • Posts: 132
Re: ERS5698 - Added new MLT and VLANS were removed from other MLTs
« Reply #3 on: January 31, 2014, 10:23:00 PM »
Thanks for the response Paul.

I made the changes from the command line using the commands indicated in the previous post.

Now that you mention it, I do recall reading somewhere that the MLT1 or the lowest MLTID is recommended for the IST. Do you think the MLT numbering would cause this type of behaviour? I always kind of thought of the MLT numbers as somewhat trivial, but then again I have no idea how the underlying switch code works.

I think I will confirm your recommendation and plan to reconfigure the switches with MLT 1 = IST ... not a big deal (other than the early morning changes, extra work, outage notifications, etc, lol)  - to me the MLT numbers are quite arbitrary … too bad they weren’t arbitrary for the switches too! If it adds stability to my core network, it needs to be done.

That being said, if anybody else has any other thoughts/opinions on what might have contributed to the mysterious VLAN deletion that I observed, I would appreciate hearing them. Has anybody ever seen this exact behavior before?

Offline Paul L

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 754
    • paulaleroux
    • Paul's Networking blog
Re: ERS5698 - Added new MLT and VLANS were removed from other MLTs
« Reply #4 on: January 31, 2014, 11:23:09 PM »
its not related to your issues. just an observation.

Nortel recommends that you used MLT1, VLAN2, and the first ports possible to make your IST. This assures that the IST would be the first interfaces to come up. We are talking about micro seconds, but they wouldn't mention it unless it was a good idea.


as for your work. when you applied the changes did it remove the vlans on both switches? 
did you type in the commands or cut and paste them?
ACSS- Avaya Enterprise Routing Switch  #8

Offline brazenhead

  • Sr. Member
  • ****
  • Posts: 132
Re: ERS5698 - Added new MLT and VLANS were removed from other MLTs
« Reply #5 on: February 01, 2014, 05:18:56 PM »
The vlans were automatically removed on both switches. It might be worth noting that I had finished applying the same commands on both switches before I noticed any issues. However, based on the time frames of reported issues, it almost seems like applying the commands on the 1st switch is what removed the vlans from both ... i.e. I think everything "went down" (unbeknownst to me) before I had a chance to apply the commands to the 2nd switch.

And actually, it seemed like I had to re-add the vlans to some of the ports twice: I re-added them on switch 1, moved to switch 2 and re-added to appropriate ports, then went back to switch 1 to verify but had to re-re-add to several ports again, etc (it seemed like the vlans were removed a 2nd or 3rd time).

I cut and pasted the commands from a "notepad" text file.

Offline Paul L

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 754
    • paulaleroux
    • Paul's Networking blog
Re: ERS5698 - Added new MLT and VLANS were removed from other MLTs
« Reply #6 on: February 02, 2014, 01:12:26 PM »
all I can say is that I wouldn't have added the VLANS to the port, then created the MLT.

I would have created the MLT, Set the SMLT, set the port tagging, then added to the VLANs to the MLT.

 
ACSS- Avaya Enterprise Routing Switch  #8

Offline shezzski

  • Jr. Member
  • **
  • Posts: 34
Re: ERS5698 - Added new MLT and VLANS were removed from other MLTs
« Reply #7 on: February 03, 2014, 02:55:37 AM »
Hi Brazenhead,

This is actually a simple mistake that gets people all the time.  There are actually errors in your applied configuration, which was

vlan members 3000 91,92
vlan members 3001 91,92
vlan members 3002 91,92


The above lines set VLAN 3000,3001 and 3002 to exist on ports 91 and 92.  This is why they were removed from the other ports/MLT's.

You should have entered the following

vlan members add 3000 91,92
vlan members add 3001 91,92
vlan members add 3002 91,92


or if you love typing you could have entered

vlan members 3000 1-24,47-48,91-96
vlan members 3001 1-24,47-48,91-96
vlan members 3002 1-24,47-48,91-96



Cheers and good luck.

Offline brazenhead

  • Sr. Member
  • ****
  • Posts: 132
*Resolved* ERS5698 - Added new MLT and VLANS were removed from other MLTs
« Reply #8 on: February 03, 2014, 07:49:40 AM »
Hi shezzski,

Thank you for helping me identify the error. Now that you have pointed out the missing add it is somewhat obvious, but I had looked at these commands half a dozen times or so and couldn't see it. I'm happy to have figured out what happened. Much appreciated!

And thanks for the other tips Paul L.

Cheers!