• November 27, 2020, 05:47:15 AM
Welcome, Guest. Please login or register. Registration is free.
Did you miss your activation email?

Author Topic: VSP7000 IST rate-limit problem  (Read 7789 times)

0 Members and 1 Guest are viewing this topic.

Offline kristjanhinn

  • Jr. Member
  • **
  • Posts: 25
    • http://ee.linkedin.com/pub/kristjan-hinn/5b/b55/a17
VSP7000 IST rate-limit problem
« on: September 11, 2014, 02:40:09 AM »
Hello,

What can cause high packet rate on IST ports? We have vsp7000 IST and edge switches connected with SMLT. First we suspected loop somewhere but we cant find any loops and SLPP does not activate also. I cannot see high packet rate on edge ports, only on IST ports. Network is up and running but sometimes vsp7000 management does not answer to ping and few packets may get lost in network. Basically network is unstable.

IST is 10Gbps and configured as MLT and VLACP is also enabled on IST ports.

Our topology is attached to the post (red circled ports have high rate).

Heres how show rate-limit looks like:
Code: [Select]
C-TUUM1-4A1#show rate-limit
Port  Packet Type  Limit          Last 5 Minutes  Last Hour  Last 24 Hours
----  -----------  -------------  --------------  ---------  -------------
1     Both         5%                 0.1%         0.0%         0.0%
2     Both         5%                 0.0%         0.0%         3.9%
3     Both         5%                 0.0%         0.0%         2.9%
4     Both         5%                 0.0%         0.0%         3.6%
5     Both         5%                 0.0%         0.0%         0.0%
6     Both         5%                 0.0%         0.0%         0.0%
7     Both         5%                 0.0%         0.0%         0.4%
8     Both         5%                 0.0%         0.0%         0.0%
9     Both         5%                 3.7%         3.4%        11.3%
10    Both         5%                 0.0%         0.0%         4.2%
11    Both         5%                 0.0%         0.0%         0.0%
12    Both         5%                 0.0%         0.0%         0.0%
13    Both         5%                 0.0%         0.0%         0.0%
14    Both         5%                 0.0%         0.0%         0.0%
15    Both         5%                 0.0%         0.0%         0.0%
16    Both         5%                 0.0%         0.0%         0.0%
17    Both         5%                 0.0%         0.0%         0.0%
18    Both         5%                 0.0%         0.0%         0.0%
19    Both         5%                 0.0%         0.0%         0.0%
20    Both         5%                 0.0%         0.0%         0.0%
21    Both         5%                 0.0%         0.0%         0.0%
22    Both         5%                 5.4%         5.6%         5.6%
23    Both         None              71.3%        23.6%        41.9%
24    Both         None              85.5%        83.2%        84.5%
25    Both         5%                 0.0%         0.0%         0.0%
26    Both         5%                 0.0%         0.0%         0.0%
27    Both         5%                 0.0%         0.0%         0.0%
28    Both         5%                 0.0%         0.0%         0.0%
29    Both         5%                 0.0%         0.0%         0.0%
30    Both         5%                 0.0%         0.0%         0.0%
31    Both         5%                 0.0%         0.0%         0.0%
32    Both         5%                 0.0%         0.0%         0.0%

CPU:
C-TUUM1-4A1#show cpu-utilization
----------------------------------------------------------------
                      CPU Utilization
----------------------------------------------------------------

Unit  Last 10 Sec, 1 Min, 10 Min, 60 Min, 24 Hrs, System Boot-Up
----------------------------------------------------------------
1          51%     43%    44%     44%     NA      44%


Btw if i set rate-limit on IST ports then management ping will go nuts. And when i disable one of the edge ports, then rate will go down. It's like behaving opposite what it should behave. Rate should go up when one of the SMLT's are disabled?

Any ideas?

Thank you
Kristjan



Offline Dominik

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1564
    • Networkautobahn
Re: VSP7000 IST rate-limit problem
« Reply #1 on: September 11, 2014, 05:20:08 AM »
Wich SW version do you run on your VSP7k ?

In the 10.3.x releases Avaya recommands to disable VLACP on the IST.
The high CPU utilization is odd, it should be around 15-18 % on avarage.
I would start to controll all SMLTs and there running state and on the Access switches the MLTs that are connected make sure that all MLts are enabled on configured correctly.

What CPu utilization do you have on the direct connected VSP7k pair ?
Do you use SPB or only classic SMLT ?

Cheers
 
Itīs always the networks fault!
networkautobahn.com

Offline kristjanhinn

  • Jr. Member
  • **
  • Posts: 25
    • http://ee.linkedin.com/pub/kristjan-hinn/5b/b55/a17
Re: VSP7000 IST rate-limit problem
« Reply #2 on: September 11, 2014, 07:11:57 AM »
SLT's seems to be working (edge switches are 1-12 ports, 22 is uplink from second vsp7k IST and 21 is not in use.
Code: [Select]
                                        SLT Info
===============================================================================
PORT  SMLT     ADMIN    CURRENT
NUM   ID       TYPE     TYPE
-------------------------------------------------------------------------------
1       71      slt     slt
2       72      slt     slt
3       73      slt     slt
4       74      slt     slt
5       75      slt     slt
6       76      slt     slt
7       77      slt     slt
8       78      slt     slt
9       80      slt     slt
10      81      slt     slt
11      82      slt     slt
12      83      slt     slt
21      84      slt     norm
22      79      slt     slt

Second vsp7k pair has low cpu usage
Code: [Select]
----------------------------------------------------------------
                      CPU Utilization
----------------------------------------------------------------

Unit  Last 10 Sec, 1 Min, 10 Min, 60 Min, 24 Hrs, System Boot-Up
----------------------------------------------------------------
1          17%     24%    19%     19%     18%     18%

i have no SPB in use all are classic SMLT's. It seems in about every hour or so the IST ports get so overloaded that VLACP packets are not moving anymore and VLACP disables IST ports. Ill try to turn VLACP off on IST ports but i dont think its the main issue here.

switch FW:
HW:06       FW:10.3.0.2  SW:v10.3.0.011
« Last Edit: September 11, 2014, 07:13:52 AM by kristjanhinn »

Offline kristjanhinn

  • Jr. Member
  • **
  • Posts: 25
    • http://ee.linkedin.com/pub/kristjan-hinn/5b/b55/a17
Re: VSP7000 IST rate-limit problem
« Reply #3 on: September 11, 2014, 08:28:32 AM »
i just discovered that if i disable bottom switch edge ports then top switch cpu will go 100% even if i only disable one port. But if i disable top switch edge ports then bottom switch cpu stays at 45%. It seems like top switch SLT is not working or it cant manage traffic at all.
« Last Edit: September 11, 2014, 08:40:46 AM by kristjanhinn »

Offline Dominik

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1564
    • Networkautobahn
Re: VSP7000 IST rate-limit problem
« Reply #4 on: September 11, 2014, 10:45:06 AM »
When you are sure that the SLT and MLT configuration is correct on your Switches
I would recommand to upgrade to 10.3.2.

Do you have enabled DiscardUntaggedFrames or FilterUnregisteredFrames on the IST Ports ?
Do you have seen something in the log files ?

The cpu spikes are really strange havenīt seen that behavior on VSP7k so far.


Itīs always the networks fault!
networkautobahn.com

Offline kristjanhinn

  • Jr. Member
  • **
  • Posts: 25
    • http://ee.linkedin.com/pub/kristjan-hinn/5b/b55/a17
Re: VSP7000 IST rate-limit problem
« Reply #5 on: September 11, 2014, 02:36:05 PM »
i'll try to upgrade today, we have FilterUnregisteredFrames enabled DiscardUntaggedFrames disabled

Offline kristjanhinn

  • Jr. Member
  • **
  • Posts: 25
    • http://ee.linkedin.com/pub/kristjan-hinn/5b/b55/a17
Re: VSP7000 IST rate-limit problem
« Reply #6 on: September 12, 2014, 02:10:00 AM »
Still no luck :(

my current topology
https://dl.dropboxusercontent.com/u/41978197/topoloogia_switchC2.JPG

both switc CPU is still around 45-50% and 10.50.8.166 CPU is overloaded to 100% time to time. Rate limit is still high on IST ports.

10.50.8.166 rate limits and cpu
Code: [Select]
C-TUUM2-4A1#show rate-limit
Port  Packet Type  Limit          Last 5 Minutes  Last Hour  Last 24 Hours
----  -----------  -------------  --------------  ---------  -------------
1     Both         5%                 0.0%         0.0%         0.0%
2     Both         5%                 1.1%         1.6%         1.6%
3     Both         5%                 3.6%         4.4%        18.1%
4     Both         5%                 2.8%         2.5%         7.3%
5     Both         5%                 0.0%         0.0%         0.0%
6     Both         5%                 0.0%         0.0%         0.0%
7     Both         5%                 0.1%         0.1%         0.3%
8     Both         5%                 0.3%         0.3%         0.3%
9     Both         5%                 1.6%         1.4%         7.1%
10    Both         5%                 2.0%         2.3%         9.9%
11    Both         5%                 0.0%         0.0%         0.0%
12    Both         5%                 0.0%         0.0%         0.0%
13    Both         5%                 0.0%         0.0%         0.0%
14    Both         5%                 0.0%         0.0%         0.0%
15    Both         5%                 0.0%         0.0%         0.0%
16    Both         5%                 0.0%         0.0%         0.0%
17    Both         5%                 0.0%         0.0%         0.0%
18    Both         5%                 0.0%         0.0%         0.0%
19    Both         5%                 0.0%         0.0%         0.0%
20    Both         5%                 0.0%         0.0%         0.0%
21    Both         5%                 0.0%         0.0%         0.0%
22    Both         None               1.8%         0.8%         1.1%
23    Both         None              93.7%        16.2%        15.4%
24    Both         None             100.0%         4.8%         4.9%
25    Both         5%                 0.0%         0.0%         0.0%
26    Both         5%                 0.0%         0.0%         0.0%
27    Both         5%                 0.0%         0.0%         0.0%
28    Both         5%                 0.0%         0.0%         0.0%
29    Both         5%                 0.0%         0.0%         0.0%
30    Both         5%                 0.0%         0.0%         0.0%
31    Both         5%                 0.0%         0.0%         0.0%
32    Both         5%                 0.0%         0.0%         0.0%

C-TUUM2-4A1#show cpu-utilization
----------------------------------------------------------------
                      CPU Utilization
----------------------------------------------------------------

Unit  Last 10 Sec, 1 Min, 10 Min, 60 Min, 24 Hrs, System Boot-Up
----------------------------------------------------------------
1          54%     51%    50%     50%     NA      46%


10.50.8.165 rate limits and cpu
Code: [Select]
C-TUUM1-4A1#show rate-limit
Port  Packet Type  Limit          Last 5 Minutes  Last Hour  Last 24 Hours
----  -----------  -------------  --------------  ---------  -------------
1     Both         5%                 0.0%         0.0%         0.0%
2     Both         5%                 0.0%         0.0%         4.2%
3     Both         5%                 0.0%         0.0%        10.3%
4     Both         5%                 0.0%         0.0%         6.0%
5     Both         5%                 0.0%         0.0%         0.0%
6     Both         5%                 0.0%         0.0%         0.0%
7     Both         5%                 0.0%         0.0%         0.4%
8     Both         5%                 0.0%         0.0%         0.0%
9     Both         5%                 0.0%         0.0%        11.5%
10    Both         5%                 0.0%         0.0%         9.3%
11    Both         5%                 0.0%         0.0%         0.0%
12    Both         5%                 0.0%         0.0%         0.0%
13    Both         5%                 0.0%         0.0%         0.0%
14    Both         5%                 0.0%         0.0%         0.0%
15    Both         5%                 0.0%         0.0%         0.0%
16    Both         5%                 0.0%         0.0%         4.2%
17    Both         5%                 0.0%         0.0%         0.0%
18    Both         5%                 0.0%         0.0%         0.0%
19    Both         5%                 0.0%         0.0%         0.0%
20    Both         5%                 0.0%         0.0%         0.0%
21    Both         5%                 0.0%         0.0%         0.0%
22    Both         None               0.0%         5.9%         5.5%
23    Both         None              29.0%        45.5%        49.3%
24    Both         None              94.9%        86.3%        31.8%
25    Both         5%                 0.0%         0.0%         0.0%
26    Both         5%                 0.0%         0.0%         0.0%
27    Both         5%                 0.0%         0.0%         0.0%
28    Both         5%                 0.0%         0.0%         0.0%
29    Both         5%                 0.0%         0.0%         0.0%
30    Both         5%                 0.0%         0.0%         0.0%
31    Both         5%                 0.0%         0.0%         0.0%
32    Both         5%                 0.0%         0.0%         0.0%

C-TUUM1-4A1#show cpu-utilization
----------------------------------------------------------------
                      CPU Utilization
----------------------------------------------------------------

Unit  Last 10 Sec, 1 Min, 10 Min, 60 Min, 24 Hrs, System Boot-Up
----------------------------------------------------------------
1          46%     50%    45%     44%     NA      45%

I enabled discarduntaggedpackets on ist ports also and upgraded FW to HW:06       FW:10.3.1.5  SW:v10.3.2.011

Im clueless :(

Offline kristjanhinn

  • Jr. Member
  • **
  • Posts: 25
    • http://ee.linkedin.com/pub/kristjan-hinn/5b/b55/a17
Re: VSP7000 IST rate-limit problem
« Reply #7 on: September 12, 2014, 03:01:12 AM »
ok now we disconnected one of the switches completely removed all SFP modules and restarted switch

on bootup we got CPU warning

## Warning:  CPU    CPLD Rev= 0D (<0x0E)

Starting Diagnostic Code Vers:  10.3.1.5

(to interrupt Diagnostics, type control-C and wait..)

7024XLS  Diagnostics 10.3.1.5

Test 101  Flash  ID Read                  - PASSED
Test 121  System SROM Labels              - PASSED
Test 127  CBM    SROM Labels              - PASSED
Test 131  DTS    Temperatures             - PASSED
Test 141  RTC    Clock                    - PASSED
Test 151  FANs   Status                   - PASSED
Test 161  Power  Supplies                 - PASSED
Test 171  POLs   Status                   - PASSED
Test 173  RDACs  Registers                - PASSED
Test 175  PPACs  Status                   - PASSED
Test 181  USB    PHY Registers            - PASSED
Test 211  TSEC   Internal Loopback        - PASSED
Test 271  Ports  Internal Loopback        - PASSED

Starting [ Agent-1 ]  Vers:  10.3.2.011

Decompressing the image  done.

Initializing ......


in total idle (no traffic no SFP's) state switch has still 40-60% cpu utilization. Second switch what is working alone atm, still has cpu 40-60%.

Offline kristjanhinn

  • Jr. Member
  • **
  • Posts: 25
    • http://ee.linkedin.com/pub/kristjan-hinn/5b/b55/a17
Re: VSP7000 IST rate-limit problem
« Reply #8 on: September 12, 2014, 03:43:05 AM »
i just made a ticke at Avaya, lets see how it goes.

Offline Dominik

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1564
    • Networkautobahn
Re: VSP7000 IST rate-limit problem
« Reply #9 on: September 12, 2014, 01:48:21 PM »
I would suggest that you have a hardware issue on your VSP7k.
In fact it has a lifetime warrenty I would ask the Avaya support for a replacement VSP.

Did you have an software exaptione ?
You can prove that with the command:
sho system last-exception

Does any of your access switches shows also high CPU utilization ?

When you have no clue at all you can try the trace command , maybe you can see something in debug massages.

Itīs always the networks fault!
networkautobahn.com

Offline kristjanhinn

  • Jr. Member
  • **
  • Posts: 25
    • http://ee.linkedin.com/pub/kristjan-hinn/5b/b55/a17
Re: VSP7000 IST rate-limit problem
« Reply #10 on: September 19, 2014, 03:57:08 AM »
Avaya is still debuging this issue. Ticket is escalated to next level. It seems like packet traffic is consuming cpu but not so much packets are realy there.

Offline Dominik

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1564
    • Networkautobahn
Re: VSP7000 IST rate-limit problem
« Reply #11 on: September 19, 2014, 04:46:25 AM »
here ist always depends wich kind of traffic do you have.
Most packets should be forwarded over the SF without hitting the CPU.
In the past I had issues with busrts of ARP packets wich will be checked by the switch even if it is running in pure L2 mode.
Could be possible that you have something similar here.

The odd think here is that yor second VSP7k pair is working normal.
Do you have the same kind of traffic/packets on your second VSP7k pair ?
Itīs always the networks fault!
networkautobahn.com

Offline kristjanhinn

  • Jr. Member
  • **
  • Posts: 25
    • http://ee.linkedin.com/pub/kristjan-hinn/5b/b55/a17
Re: VSP7000 IST rate-limit problem
« Reply #12 on: September 19, 2014, 08:16:59 AM »
these ARP requests should be seen with wireshark right? We have sniffed all vlans on that switch and found nothing abnormal. Avaya has seen those logs also.

Offline alekm

  • Rookie
  • **
  • Posts: 9
    • alekmurray
Re: VSP7000 IST rate-limit problem
« Reply #13 on: September 19, 2014, 07:44:56 PM »
Do you see the same problem when you unplug one link on the IST? 


Offline kristjanhinn

  • Jr. Member
  • **
  • Posts: 25
    • http://ee.linkedin.com/pub/kristjan-hinn/5b/b55/a17
Re: VSP7000 IST rate-limit problem
« Reply #14 on: September 20, 2014, 03:33:07 AM »
Do you see the same problem when you unplug one link on the IST?

yes, actually we have only one ist peer online atm, its more stable with one peer atm.