New ERS5000 Software v6.2.4

[Michael McNamara]

What we notice is that despite the fact that we run a really really simple configuration (layer 2 only) the units in the stack constantly operate on 95% memory used. Administrating them over JDM was clumsy and really really really slow.

It has gotten a bit better with EDM, since...well...that thing loads slow anyway.

What version of software are you running today?

[Dave]

What we notice is that despite the fact that we run a really really simple configuration (layer 2 only) the units in the stack constantly operate on 95% memory used. Administrating them over JDM was clumsy and really really really slow.

It has gotten a bit better with EDM, since...well...that thing loads slow anyway.

What version of software are you running today?

6.2.3

Only remaining issue at the moment is the random base unit reboots that some stacks keep having.

Memory usage is still alarmingly high. All units in about 20 stacks have <15mb memory unused and I have no idea why.

[Michael McNamara]

6.2.3

Only remaining issue at the moment is the random base unit reboots that some stacks keep having.

Memory usage is still alarmingly high. All units in about 20 stacks have <15mb memory unused and I have no idea why.

The reason the base unit reboots is because it's acting as the brains for the entire stack. So any software exceptions that the stack experiences will result in the base switch rebooting itself (to recover). If you made the second switch in the stack the base you'd find that the problem will generally follow whichever switch is base.

You can occasionally catch the exception and crash by attaching a console (with buffer) to the serial port of the switch. If your console doesn't have a buffer you'll need to attach a serial printer to try and capture the output.

I'm still running 6.1.5 software as my preferred code although I'm getting ready to start heavily testing 6.2.4 software. I have hundreds of large stacks (6-8 switches) running 6.1.5 in a Layer 2 only configuration and they are very reliable.

If the management interface of the switch/stack isolated from the user traffic? Do you have management VLAN? I've seen similar problems when the management interface of the switch is mixed with user traffic. Why? I would guess that there's some traffic being generated by the user devices that is creating issues for the switch management interface. I've seen this with IGMP, Multicast and broadcast traffic.

Looking at the 6.2.4 software release notes I see two (resolved) software exceptions that were caused by external traffic to the switch.

Good Luck!

[Dave]

6.2.3

Only remaining issue at the moment is the random base unit reboots that some stacks keep having.

Memory usage is still alarmingly high. All units in about 20 stacks have <15mb memory unused and I have no idea why.

(...)
If the management interface of the switch/stack isolated from the user traffic? Do you have management VLAN? I've seen similar problems when the management interface of the switch is mixed with user traffic. Why? I would guess that there's some traffic being generated by the user devices that is creating issues for the switch management interface. I've seen this with IGMP, Multicast and broadcast traffic.
(...)

Thanks for the info.

How did you prove/figure out if user generated traffic was "molesting" the management interface of the switch?

That could indeed be one of our problems and if there is a way to prove it, I might be able to finally steer the customer to a more "best practice oriented" approach...

[Dominik]

@Dave

To prove that you have some options:

-Mirror one of the Uplink Ports and inspect the traffic
-enable IPFIX to a Flowanalyzer
-check the ports statistics

As Michael mentioned I would recommand to seperate the management of the switch to a seperated VLAN so that the switch management IP receive less broadcast/multicast packets that must be proceeded by the switch cpu.

Good Luck

[Dave]

@Dave

To prove that you have some options:

-Mirror one of the Uplink Ports and inspect the traffic
-enable IPFIX to a Flowanalyzer
-check the ports statistics

- enabling IPFIX is not an option here, because collecting data who communicated when with who is a huge lawsuit filled pandora's box in this company...(or in germany in general...)
- port statistics are not showing any dropped packets or anything unexpected.

looks like only solution left is to mirror the uplinks and inspect the capture files....

As Michael mentioned I would recommand to seperate the management of the switch to a seperated VLAN so that the switch management IP receive less broadcast/multicast packets that must be proceeded by the switch cpu.

Good Luck

Yeah...that would be a nice solution, but its hard to get this particular customer to touch his "tried and tested" setup, even though its not as stable as it should be...

[AVAGO]

Upgraded 4 6 unit stacks over Sat & Sun, all went well no issues, However on just one unit in one stack the switch ( 6 in this case ) has the power light on and just the status light flashing, all other units can see the stackup and stackdown.  all seems to be working OK, has anyone else running 6.2.4 had simlar issue, it could be just coincident, The unit was reset but still the same, maybe its about to fail,

I'll get the car now !

[Dave]

Upgraded 4 6 unit stacks over Sat & Sun, all went well no issues, However on just one unit in one stack the switch ( 6 in this case ) has the power light on and just the status light flashing, all other units can see the stackup and stackdown.  all seems to be working OK, has anyone else running 6.2.4 had simlar issue, it could be just coincident, The unit was reset but still the same, maybe its about to fail,

I'll get the car now !

I upgraded one live 8 unit stack and its been working for a week without any issues.

I will upgrade another 20 8 unit-stacks on saturday. If I run into any issues I will be posting after they are resolved....

[Straphlinger]

We've just finished an upgrade of 288 stacks of 5500's to 6.2.4. No problems yet...
The upgrade went very smoothly using CLI manager and some basic scripts( From 6.2.1 to 6.2.4). None of the stacks failed to return after the reboot. And it appears that all the settign have come across. It even retained the QOS settings, which were lost every reboot on 6.2.1.

The only issue we  have is with our 5632's.
They have been rebooting after the following error,
Sw Exception: Task tDHCP, Type Data Access, PC 0x00ff24b0, SP 0x07860ea0
This problem occurs on 6.2.1,6.2.3 and 6.2.4.
Avaya is investigating and confirmed that they have identified the problem and are working to have it included in 6.2.5.

[Michael McNamara]

We've just finished an upgrade of 288 stacks of 5500's to 6.2.4. No problems yet...
The upgrade went very smoothly using CLI manager and some basic scripts( From 6.2.1 to 6.2.4). None of the stacks failed to return after the reboot. And it appears that all the settign have come across. It even retained the QOS settings, which were lost every reboot on 6.2.1.

The only issue we  have is with our 5632's.
They have been rebooting after the following error,
Sw Exception: Task tDHCP, Type Data Access, PC 0x00ff24b0, SP 0x07860ea0
This problem occurs on 6.2.1,6.2.3 and 6.2.4.
Avaya is investigating and confirmed that they have identified the problem and are working to have it included in 6.2.5.

That's an impressive number of switches/stacks... hope you scripted that upgrade!

I just recently deployed 10 ERS 5632s running 6.2.4 software as part of our 10GE MAN redesign. We're only doing Layer 2 with MSTP across 2 VLANs and so far so good.

Cheers!

[Bez]

Hi all,

Are there any more recent experiences with 6.2.4 ?

I'm looking to upgrade some 5510s running layer 3 to enable us to run PIM-SM for SIP Dect

Cheers

Bez