Replace IST/SMLT configuration with Normal MLT/spanning tree

[paquetteg1]

Hi,

We have a couple of ERS 8600 configured with a IST/SMLT setup each 8600 has 3 x 8616SX cards (16 1GB SX fibers),
there are around 1500 IP/mac addresses in the arp cache at any given time and the software code is 3.7.9.
Each 8600 has 35 trunks feeding 35 bs-470/bs-5510 Nortel stacks. 6 IP interfaces on each 8600 define the IP addresses assigned to 6 vlans (with VRRP), all the vlans are pushed to the 35 trunks The IST is made of 3 x 1GB fibers.
For a reason that I will not explain we have to change the IST/SMLT configuration to a normal MLT with spanning tree configuration with the STG of one 8600 at a lower priority than the other 8600.
My question is, is there a way to GRADUALLY ACCOMPLISH this besides stopping everything, changing the IST MLT configuration to a normal MLT configuration and enabling STP, changing the configuration of the 35 SMLT's to normal MLT's and enabling spanning tree and restart everything?
I would very much appreciate ideas/opinions on the subject.

Tahnks in advance,

Gaetan

[Michael McNamara]

Hi Gaetan and welcome to the forums!

The obvious question many of us will have is "what are you trying to accomplish with your design change?" The design you're trying to implement is one that I have personally left behind many years ago and hope to never visit again anytime soon. I've seen so many overly complex STP and RSTP configurations that I don't ever care to go back.

Without thinking too hard the answer to your question is NO, you can't implement such a change gradually -- or at least it isn't one that I would personally try to implement. You could certainly do a BIG BANG approach but I don't blame you for wanting to avoid that like the plague. You could implement a gradual conversion to MLT from SMLT but you'd still need to leave STP behind. With an MLT approach you'd need to terminate both uplinks for every edge switch on a single ERS 8600 switch. The other issue you'll probably run into is the MLT limit in 3.x software (very old software by the way). I believe you are limited to just 32 MLT groups/trunks. So if you wanted to convert from SMLT to MLT you'd need to connect some switches to one core switch and then others to the second core switch. Leaving you without redundancy in the case of a single core switch failure.

Perhaps you could tell us what's driving you to such a dramatic change and we could provide you other options. I suspect perhaps you're having support issues and are hoping for an easier and simpler design?

If on the other hand you are looking to change vendors I would suggest you stand-up a new core network and gradually migrate your edge switches to that new core. You could setup a LACP/802.3ad link between the two cores and gradually migrate the IP interfaces, routing, etc.

Let us know.

Cheers!

[qazzie]

Frankly I wasn't really wanting to post a reply on this thread. Since Michael explained the thoughts behind the proposed change allready.

It's ok to talk about changing from Nortel to juniper,cisco,hp or whatever even on this forum But even then I would go for another approach, so without any background info we can only guess the intentions.

q

[paquetteg1]

Hi again,

Thanks for the quick replies, I'm glad I found this forum on Nortel equipment and reading throuh some of the topics I can tell that there are many knowledgable people on the subject.
To get back to the topic I submitted, Michael mentioned "I suspect perhaps you're having support issues and are hoping for an easier and simpler design?" that's exactly it! We experienced many network faillures through the years from which 95% were caused by human error (wrong configurations, wrong connections...etc) as you all know a SMLT setup is very unforgiving with mistakes since there is no STP to protect you. It is also more complicated to troubleshoot when "normal" problems occur not to mention the difficulty to find knowledgable people on this type of network.
The local management has asked us if we could "simplify" the complexity of the network and protect it against unwanted loops by getting rid of SMLT and reintroducing STP and accomplishing this without any "down time"  My answer was NO! but I was asked to ask for second opinions hence this topic.
Thanks again for the replies and I'll let you know soon how it unfolds.

Best regards,

Gaetan

[Michael McNamara]

Hi Gaetan,

There are a ton of features available that can help you protect the network...

Have a look at this blog post and the follow-up comments; http://blog.michaelfmcnamara.com/2007/10/nortel-ers-5520-pwr-switch/

Here's a quick overview of the features that will help protect the network;

• run STP on all edge ports (just not on the uplinks/downlinks)
• run BPDU-filtering on all edge ports (prevent users from connecting their own switches)
• set rate-limiting on all edge switches to 5%
• run CP-Limit on the core to identify and contain broadcast storms/loops
• run VLACP on uplinks/downlinks to identify FEFI issues when auto-negotiation is not available
• run SLPP on the core to identify misconfigured MLT links and shut them down

Those are just a few quick ideas that will generally save the day (they have for me on numerous occasions).

I would also suggest you review the technical guides that Nortel has released. They review all of the recommendations I've made above in great detail. I can provide the links if you can't find them.

There are quite a few folks here more than happy to help you.

Please let me us know how you make out.

Good Luck!

[paquetteg1]

Hello Again,

Good news, looks like smlt/ist is staying. The deciding factor was explaining to the local management the time it takes to recover from a link faillure between smlt and mlt/stp, since there is a lot of "real time" applications going on on our network (medical environment) the sell was easy.
The first thing we want to do is to implement VLACP on our smlt links (as Michael suggested in the previous reply) , we had a few instances where a GBIC failled on a 5510 stack (autoneg. disabled) and the port on the 8600 end stayed up and it kept pumping data on the dead link with disastrous results.
Ufortunalely the sofware versions of our stacks don't support VLACP, we have 3.5.0.58 on our BS470 stacks and 4.0.1.32
on our BS5510 stacks, according to Nortel 802.3ad/vlacp guide we need at least 3.6 for the 470's and 5.0 on the 5510's.
What are the best versions to use here for "bug free" VLACP? and what are the upgrade paths from the versions we have? BTW our 8600's run 3.7.9 (VLACP support) both 8600 need extra memory to uograde to 4.x.
One more question regarding one of Michael's recommendations in the previous reply "set rate-limiting to 5% on all edge switches", with multicast trafic limited to 5% is it still a good idea to turn on igmp snooping on edge switches?

Thanks again for you help/opininions it is very much appreciated, I'll keep updating this topic if you don't mind.

Gaetan

[Michael McNamara]

Hi Gaetan,

I think you've made a good decision. You can still get a lot of life out of the hardware/equipment you have. You might want to investigate putting the equipment under a 1 year maintenance contract to get the software upgrades.

If you have ERS 5500/5600 series switches I would strongly suggest that you use Autonegotiation on the uplinks to the ERS 8600 core. We use autonegotiation where possible (supported) and we also use VLACP everywhere. The legacy 470 switches don't support autonegotiation so VLACP is critical, while the newer 470-PWR (PoE) switches support autonegotiation (newer Broadcom chipset).

I would advise a minimum of 3.6.8 for your 460/470 switches, and 5.1.5 for your 5500 series switches. I would suggest 4.1.8.3 software for your ERS 8600 switches to start, although you may want/need to upgrade them to 5.1.1.1 eventually should you want support from Avaya.

I'm currently using 3.7.2 and 3.7.4 for 460/470s and I'm slowing migrating from 5.1.2/5.1.5 to 6.1.2 on the 5500 series switches.

We've have IGMP snooping disabled for years here... IGMP Snooping is now disabled in a factory default configurations (that was changed years ago after all the issues with IGMP Snooping). I would suggest you leave it off unless you have some serious Multicast applications/traffic.

You should be very careful in how your upgrade your switches, since the code you are running is so old you'll probably need to perform quite a few interim upgrades - you won't be able to upgrade straight to the latest code from the code you are currently running.

Good Luck!

[qazzie]

 

IGMP Snooping is now disabled in a factory default configurations (that was changed years ago after all the issues with IGMP Snooping)

bringing up old memories huh? :-)

Regarding VLACP there is something in depth you might would like to know. Don't want to make it to complex for you, but for every stack you connect on the ERS8600's you should use different ethertype's. There is a list of ethertypes which -preferably- should not be used since they are used for other protocols. And having more than 30 stacks you might run into those. I haven't access to my info to that so I will post that later. And also in the Nortel documention they mention the use of a certain mac-adres you should allways use. It's a hardcoded mac-adres. So in case of malfunction of the connected switch/stack it won't traverse past the ingress port queue.

I found the list
Ethertype Protocol
0x0000-
0x05DC
IEEE 802.3 length
0x0800 IP, Internet Protocol
0x0806 ARP, Address Resolution Protocol
0x8035 DRARP, Dynamic RARP. RARP,Reverse Address Resolution Protocol
0x80F3 AARP,AppleTalk Address Resolution Protocol
0x8100 EAPS,Ethernet Automatic Protection Switching
0x8137 IPX, Internet Packet Exchange
0x814C SNMP, Simple Network Management Protocol
0x86DD IPv6, Internet Protocol version 6
0x880B PPP, point-to-Point Protocol
0x880C GSMP, General Switch Management Protocol
0x8847 MPLS, Multi-Protocol label Switching(unicast)
0x8848 MPLS, Multi-Protocol label Switching(multicast)
©2009 Nortel (9-Feb-2009) Page 75 of 106
0x8863 PPPoE, PPP over Ethernet(Discovery Stage)
0x8864 PPPoE, PPP over Ethernet(PPP Session Stage)
0x88BB LWAPP, Light Weight Access Point Protocol
0x8E88 EAPOL, EAP

You will use short timers on access stack so increase the time-out scale to 5. Standard is 3 I believe, it's just some added safety in hairy situations whereas these are better in-the-field-timers.

cheers
q

[Michael McNamara]

Don't get me started with IGMP Snooping, that feature was so bug prone it was just plain dreadful.

There was an issue a long time ago with SLPP and VLACP I believe because they both used the same Ether type. I'm pretty sure in involved SLPP, it was pretty funny because the second you enabled SLPP it would start going crazy because whatever protocol/feature was using the same Ether type would set it off.

I use the standard MAC address and short timers at 500ms with 5 retries. I only use different MAC addresses when running VLACP on an IST link between two ERS 8600 switches along with long timers - it's in the network design guidelines somewhere, can't quite remember where just right now.

For everyone else following the thread, just in case your not following his rationale, Qazzie is protecting against mistakenly connecting the wrong edge/closet switch to the wrong downlink port(s) at the core. By utilizing unique Ether types for each edge/closet switch, VLACP will not come up unless the Ether types at both ends match. This guarantees that you can't mistakenly connect the wrong core ports to an edge/closet switch. This would be especially problematic if say someone connect one leg of an SMLT/SLT to one edge/closet switch and the second leg of that same SMLT/SLT to a completely different edge/closet switch.

That configuration is a little extreme in my opinion but it certainly guarantees that no one will be able to screw it up.

Thanks for the feedback Q!

[paquetteg1]

"This would be especially problematic if say someone connect one leg of an SMLT/SLT to one edge/closet switch and the second leg of that same SMLT/SLT to a completely different edge/closet switch."
It's been done a few times at my site, this is when 2000 users start chasing you with sharp instruments! :-)

Regards,

Gaetan

[paquetteg1]

Hi,

Thought I'd let you know what's going on at our site lately.

- We upgraded all our BS-470 stacks from 3.5.0.58 to 3.7.1.8 in multiple steps and
  enabled VLACP on both ends of our smlt links to the 8600's

- We upgraded all our BS-5510 stacks from 4.0.1 to 5.1.5.8 in mutiple steps and
  enabled autonegotiation on both ends of our smlt links to the 8600'S
  (we use to have autonegotiation disabled before)

I have a few points that I'm still not clear about and I'd like to ear comments from this forum:

1- My understandind is that if you have autonegotiation support between the 5510's
   and the 8600's you don't really need VLACP since autoneg will take care of disabling
   the port on a faulty link (how fast will it do it conmpared to VLACP?)

2- IST between our two 8600's is made of 3 1GB fiber links with autonegotiation disabled.
   What is better, leave autoneg disabled and enable VLACP or enable autoneg and not VLACP?
   If I enable autoneg on both ends the IST can I do it on the fly without too much disruption?

Looking forward to ear from this group,

Thanks in advance to all

Gaetan

[Michael McNamara]

I have a few points that I'm still not clear about and I'd like to ear comments from this forum:

1- My understandind is that if you have autonegotiation support between the 5510's
   and the 8600's you don't really need VLACP since autoneg will take care of disabling
   the port on a faulty link (how fast will it do it conmpared to VLACP?)

2- IST between our two 8600's is made of 3 1GB fiber links with autonegotiation disabled.
   What is better, leave autoneg disabled and enable VLACP or enable autoneg and not VLACP?
   If I enable autoneg on both ends the IST can I do it on the fly without too much disruption?

Looking forward to ear from this group,

It's actually poorly documented that VLACP does much more than just resolve FEFI when you don't have autonegotiation enabled. You should always used autonegotiation where available along with VLACP. The use of VLACP provides a facility (beyond just link) for Nortel/Avaya to pause traffic to/from various ports while the network is recovering from an outage or reset, etc. VLACP is heavily used in a SMLT environment providing a means of allowing the core switch to get up and going (IP routes, OSPF neighbors, forwarding tables, etc) before allowing an edge switch to start forwarding frames up the uplink.

I personally have a lot of Ethernet Switch 470s which don't support autonegotiation so VLACP also prevents FEFI situations where one side of the link remains up while the other side is dead and the switch ends up forwarding traffic to a black hole.

With regard to your IST ports you should enable VLACP with long timers (not the short timers you use with all your edge switches) then you should admin-down the ports (one at a time) as you enable autongeotiation. You should be able to admin-down the ports one at a time and make the configuration change without any impact to your network.

Good Luck!

[paquetteg1]

Thanks for the quick reply Mike,

I Just did enable VLACP on the 5510's, I had a little suprise doing so. Contrary to the 470's
which allowed for setting up VLACP in advance on the links (links would stay up) and then
doing the 8600 side, the links came down as soon as I setup VLACP on the links on the 5510 side, I then had
to rush to do it on the 8600 end to bring the links back up under VLACP control. I was relying on the documentation
that says that enablind VLACP on the 470's, 4500's and 5500's links first would not bring the them down.

Regarding the IST, if I admin down one of the 3 links don't I take the risk of bringing the whole IST down
since these links are part of an MLT. Can you reassure me on this...

Thanks again for your precious help.

Gaetan

[Michael McNamara]

I Just did enable VLACP on the 5510's, I had a little suprise doing so. Contrary to the 470's
which allowed for setting up VLACP in advance on the links (links would stay up) and then
doing the 8600 side, the links came down as soon as I setup VLACP on the links on the 5510 side, I then had to rush to do it on the 8600 end to bring the links back up under VLACP control. I was relying on the documentation that says that enablind VLACP on the 470's, 4500's and 5500's links first would not bring the them down.

In older versions of software you had 2.5 seconds (depending on the timers configuration ~ 500ms * 5 retries = 2500ms) to get both sides configured before VLACP would knock the port down. Avaya/Nortel changed the behavior of VLACP recently so that when enabled it automatically goes into a down state until VLACP establishes a connection with the remote switch. Originally, the port would continue to function until the VLACP timers expired giving you a small window to quickly enable the other side of the link.

The information above is covered in the software release notes for each switch... always important to review all the release notes, especially if you are jumping versions - you still need to read all the release notes to understand all the changes that have been made since the older software releases.

Regarding the IST, if I admin down one of the 3 links don't I take the risk of bringing the whole IST down since these links are part of an MLT. Can you reassure me on this...

I can assure you, that if you admin-down just one port your IST/MLT will continue to function (assuming of course that the other ports are working properly). You can go all the way down to a single port on each ERS 8600 and the IST/MLT will continue to function.

It's identical to any MLT. If you have a two port (four ports looking at both switches) you can disable a single port/link the other port/link will still carry the traffic and the switchover is essentially instantaneous. If anything you may notice a single ping [packet] loss at the time.

I can understand your concern and hesitation. Why not schedule a downtime window (off-hours if necessary)... if you end up not taking any downtime then you look like a hero. On the other than if you have issues then you're covered and no one should be looking to hang you from the light post.

Good Luck!

[paquetteg1]

Hi all,

I just like to take a moment to thank this forum (Mike in prticular) for the tremendous help that was provided to us.
We managed to complete the upgrade work with success and convince the management to stay with the
SPLIT MLT feature.
On another topic, does this forum deal with the Nortel wlan security circa 2270 now discontinued?
If yes, should I open a new topic?

Thanks again and best Regards,

Gaetan