Software Levels Across an IST on 5632's

[Telair]

I have found two stacks of 5632's connected to each other via an IST link ( 2x10Gig links ).  The thing that makes me go "humm..." about this is that it seems the switches on both sides of the link are running different version of software.  One side is running v6.2.0.201 and the other side is running v6.1.2.501.  I
recall being yelled at by an instructor a long time ago never to run different code levels in an IST cluster with 8600's, but does that extend to 5632 switches as well?

[Flintstone]

Hi Telair,

I agree that you should have the same software on both sides of the IST links, but I have had different versions of software on both sides of the IST with no issues with our ERS 5600s.

CheerZ

[Telair]

OK.  Humm...  I am seeing a lot of "FrameTooLongs" errors in JDM counting up on some ports connected to some NAS/SAN boxes and then counting up a lot on the IST between the 5632's.  Seems to be incrementing at about 32,000 packets per second.  The number of "FrameTooLongs" seems to roughly be the same as "Oversized Packets" if I look at the port stats via the telnet text menu.  I was just wondering if mismatching software on the switches could be causing issues.  Both stacks have Jumbo Frames enabled.

[Michael McNamara]

It would certainly be a "best practice" to keep them at the same version... unless you are eager to provide to-do interoperability testing for Avaya on your dime.

Cheers!

[Dominik]

Hi Telair,

I was also suprised that ERS5xxx swithces work together as a switchcluster with different SW versions.
On the IST implementation at the ERS5xxx Avaya hadnīt change so much, as on the ERS8600.
I was recently upgrading some ERS5520 switch cluster from SW 5.x to 6.2 and here also the IST had come up with a pair of 5.x and 6.2 SW.
I had only the constelation for a few minutes, until I upgraded the second switch cluster pair.

With the frames to long , I would investigate if anywhere tagged frames where send to accesports.
You can try to activate on your uplinks the following  features:
discardUntaggedFrames
FilterUnregisteredFrames


Good Luck

[mohan]

Hi everyone,

We have a strange problem regarding 5632's.Our scenario is all 5698 switches were connected to 5632's in SMLT architecture.Users connected to 5698's were getting intermittent ping responses. After troubleshooting we found that in Both 5632's which were connected through IST(Two 10gis's)  SMLT running type was normal(should be SMLT).We disabled all the links in one of the 5632 for traffic to move on. After some troubleshooting with Avaya Engineer, we were requested to upgrade from 6.1.2.029 to  6.2.2.25,we were told that this was a known issue.
Does anyone seen this type of problem??(suddenly SMLT running type became normal from SMLT).

In the upgrading process we upgraded one of the switch. Now  a new problem popped up,the upgraded switch was not accessible from Management stations in other subnets.We changed ip mgr in the switch through console,we are managing this switch by telnet into peer switch and then telnet into problem switch.

[Dominik]

Hi mohan,

I also had a lot of strange problems with SMLTs and ERS5632 /SW6.1.x). I also updated the switches to SW 6.2.2 , wich fixed all my SMLT problems
so far. Running two ERS5632 switchclusters with SW 6.2.2, after 2 month it looks like all SMLT problems are fixed.

With management access I had no problems during the update,  but you have to use the EDM , in fact of JDM is no longer supported since SW6.2
For that you have to aenable Http web access on the switch to use the EDM.

Can your management station reach the IP interfaces of the ERS5632 with icmp ?
Maybe your problem is realted to the ERS5632 that still runs SW 6.1.1 , I would try to shut down the ERS5632 with the old software and test the connectivity.
I think an update on the second ERS5632 to 6.2.2 should be done anyway.

Good Luck

[Telair]

I'll have to leave the 5632's software alone for now until we get a system outage window and can make both switches the same going forward.  Just hope nothing happens between now and then that I need to talk to Avaya support for. 

As for the "FameTooLongs" errors, I got one of the NetApp guys to take a packet trace from the filer.  When I load it in Wireshark it shows a lot of packets that are 9018 bytes in size when the document I found from Avaya says the 5000 series switches can handle a maximum of 9000 byte frames.  Anyone run in to this before?

[Telair]

After much wailing and gnashing of teeth, we got Avaya to tell us that the ERS5000's have a jumbo frame MTU of 9216 bytes.  Not the 9000 they list in their documentation or the 9018 the support desk told me.  That tidbit should help someone else out in the future.

[Uncle John]

Sustain & testing only tests interoperability between to switches connected via IST for 24 hours running different versions of code.  It is not tested beyond 24 hours.  The reason why we support this is for "Hitless Upgrades".  If both switches are 5.0 code and you want to upgrade to 5.1, simply disconnect one switch in the IST pair.  The traffic via SLT/SMLT will redistribute their load over to the other switch.  Upgrade that switch, then reconnect it.  The IST will re-form.  One switch is now 5.1, the other 5.0.  The SLT/SMLT's will redistribute their load across both switches.  Then disconnect the other switch.  Upgrade it.  Reconnect.  There you go.  We do this in production at hospitals/etc... all of the time.  It does work well.  READ THE RELEASE NOTES as there are a couple of issues with certain versions of code!  Try it first until you are comfortable with it.  Chrysler corporation's robotic production line runs on 8600's paired with ISTs.  They told me that it would cost them $1,000,000.00/minute for an outage.  I had to do an upgrade during production, it worked!

Cheers, Uncle John.