Issue with STP over LACP trunk and Juniper Ex

[pw]

Hey,

being in the same position as scmcdonald - we used to be a Nortel only Shop but since things are getting worse these days - we'll update our core with a large Juniper VC (EX4500).

I lately spend some time testing LACP between ES470 an the VC.
One thing I noticed is that, because the LAG/MLT is dynamic, every time
the connection between the ES470 an the VC recovers after a disruption
spanning tree is (magically) active again.
This is quite annoying since the config on the ES470 changes without any intervention.

On the other side we saw some problems with static configured MLT's where the gbic
did not work correct (no VLACP was not configured - not an option in a multi vendor env).

Did anyone of you manage disable stp permanently?

@Michael: Thanks for your very useful website and the forum!

Regards
Philipp

[Dominik]

@Phillipp

I disabled STP on all Uplinks and only use STP as a Loop protection on the access ports.
Works well for me, for preventing Loops on the uplinks I use SLPP on the Avaya equipment.

I have only expierence with Juniper SRX devices but on these devices I had some strange issues
with LinkgAggregation. Is it better on the EX4500 series switches ?

Do you use static (MLT) link aggregation to connect your JunOS devices with your Avaya switches or dynamic (LACP) ?

Cheers

[pw]

Hey Dominik,

I'm using dynamic LAG's (LACP) on the ES470 Uplinks - following NN48500-502 v4.7

  interface fastEthernet 25,26
  lacp key 5
  lacp mode active
  lacp timeout-time short
  lacp aggregation enable

  spanning-tree port 25,26 learning disable


  RES-24-01#show lacp port  25,26
                                    Admin Oper         Trunk Partner
  Port Priority Lacp    A/I Timeout Key   Key   AggrId Id    Port    Status
  ---- -------- ------- --- ------- ----- ----- ------ ----- ------- ------
  25   32768    Active  A   Short   5     16389 8193   6     5       Active
  26   32768    Active  A   Short   5     16389 8193   6     9       Active


  RES-24-01#show spanning-tree port 25,26
  Port Trunk   Participation   Priority  Path Cost    State   
  ---- -----  ---------------  --------  ---------  ----------
  25   6      Disabled
  26   6      Disabled

Now shutdown (on the VC side) or unplug one Uplink Fiber.


  RES-24-01#show lacp port  25,26       
                                    Admin Oper         Trunk Partner
  Port Priority Lacp    A/I Timeout Key   Key   AggrId Id    Port    Status
  ---- -------- ------- --- ------- ----- ----- ------ ----- ------- ------
  25   32768    Active  A   Short   5     16389 0            5       Active
  26   32768    Active  A   Short   5     16389 0                    Active


  RES-24-01#show spanning-tree port 25,26
  Port Trunk   Participation   Priority  Path Cost    State   
  ---- -----  ---------------  --------  ---------  ----------
  25          Disabled
  26          Disabled


Reactivate the Link

 RES-24-01#show lacp port  25,26       
                                  Admin Oper         Trunk Partner
  Port Priority Lacp    A/I Timeout Key   Key   AggrId Id    Port    Status
  ---- -------- ------- --- ------- ----- ----- ------ ----- ------- ------
  25   32768    Active  A   Short   5     16389 8193   6     5       Active
  26   32768    Active  A   Short   5     16389 8193   6     9       Active


  RES-24-01#show spanning-tree port 25,26
  Port Trunk   Participation   Priority  Path Cost    State   
  ---- -----  ---------------  --------  ---------  ----------
  25   6      Normal Learning  128       1          Listening
  26   6      Normal Learning  128       1          Listening

In the moment the LAG is up again - STP kicks in and gives you a nice 30 sec timeout :-(

I agree that the SRX (still) has some sharp edges, the EX4500 with 11.4 works stable (so fare).

I'll try to validate my observation with ERS5510 and ERS8300.

Have fun!

[KT]

I would also recommand to use a MLT instead of an LACP Trunk.

Cheers!!!!

[Michael McNamara]

I would also recommand to use a MLT instead of an LACP Trunk.

Unfortunately MultiLink Trunking is a Nortel/Avaya proprietary (static) trunking protocol which is not available in Juniper's Ethernet switching products so @pw needs to utilize an IEEE standards such as 802.3ad or LACP.

How does the Juniper handle BPDUs over multiple links in a LAG? Does the Juniper transmit BPDUs over all links or just the first?

You should be able to get this to work if you want Spanning Tree running between the switches but there are probably a few things to need to do.

If you just want to remove the whole Spanning Tree issue DO NOT want to disable Spanning Tree globally, just disable it at the physical port. If you disable Spanning Tree globally the switch will propogate BPDU frames. If you have Spanning Tree enabled globally but disabled at the physical port it will drop any BPDU frames it recieves.

What does is the Juniper running? RSTP, MSTP?

You probably need to configure the Avaya Ethernet Switch 470 for either RSTP or MSTP (requires reboot) so it matches with the Juniper (they are backwards compatible but best to eliminate any potential issues. I'm also not sure how the Nortel/Avaya switch handles BDPUs on LACP links. On a MLT (by default) the Nortel/Avaya switch will only transmit BPDU frames on the lowest number interface. You need to explicitly enable it on an MLT with the following, "mlt 1 bpdu all-ports". I'm not sure if there is a similar command for an LACP configuration.

You might look at this for some ideas;
http://blog.michaelfmcnamara.com/2011/06/avayas-multilink-trunk-and-spanning-tree-protocol/

Cheers!

[TankII]

Is there an equivalent Juniper command for Cisco's GEC?  That's a static 802.3AD configuration which the Nortel/Avaya MLT is compatible with.  We tested with with EX4200's though.

TankII

[pw]

Morning,

regarding the BPDU's - Junos sends BPDU's over all links that are member of a LAG.

My ultimate goal is to replace our core (ERS8600) as well as the distribution layer (ERS1600)
with a large VC. All access switches will be directly connected to the VC - STP
is only used on access-ports in order to prevent loops.
Routing is done in our central Firewall.

I did some quick tests with the VC and ERS8300 (standalone and switch-cluster)
and did not notice any problems with STP once the 2nd (LACP) Link was reconnected.
For me it looks like this is a bug in the ES470 firmware (v3.7.6).
(I'll try to open a ticket, but since we already have quite a few I don't expect much)

Just for reference I'm quoting the Juniper Docu for the cons of static LAG's.

• There is no keepalive mechanism to determine whether the entire path is up if there
is an intermediate device between the bundle.
• There is no way to ensure that the port physically connects to the correct device
• There is no way to ensure that the configuration is correct on both sides.

@Tankll A static LAG is a ae without "aggregated-ether-options lacp" configure ...

cheers

[Michael McNamara]

I'm not entirely sure that I'm understanding the issue with the 470. Are you saying that with STP disabled on ports 25,26 and if you disconnect one of the interfaces in the LAG then STP get's enabled on the ports on the 470?

Cheers!

[pw]

What I was trying to say:

- create a LACP LAG where STP is disabled
- remove on Link (Cable) of the LAG -> STP disabled
- reconnect the Link (Cable) -> STP is active

Thanks for your time

[Michael McNamara]

That's a very interesting issue... shouldn't be hard to test and duplicate.

I have access to a Ethernet Switch 460 (v3.7.6) that I can test with, I'll set it up later and post the results.

[Michael McNamara]

I've split the thread from @Phillipp's original post and renamed the topic.

[Michael McNamara]

I just setup a quick lab between an Ethernet Routing Switch 5520 and a Ethernet Switch 460 and I was unable to reproduce your result.

I setup an LACP trunk group on ports 23 and 24 between the two switches. I used key 20 for the ERS 5520 and I used key 10 for the 460.

5520-48T-PWR(config)#show lacp port 23,24
                                  Admin Oper         Trunk Partner
Port Priority Lacp    A/I Timeout Key   Key   AggId Id    Port    Status
---- -------- ------- --- ------- ----- ----- ----- ----- ------- -------
23   32768    Active  A   Short   20    8212  8224   32    23      Active
24   32768    Active  A   Short   20    8212  8224   32    24      Active

5520-48T-PWR(config)#show spanning-tree port 23,24
Port Trunk   Participation   Priority  Path Cost    State
---- -----  ---------------  --------  ---------  ----------
23   32     Disabled
24   32     Disabled

And the Ethernet Switch 460...

460-24T-PWR(config)#show lacp port 23,24
                                  Admin Oper         Trunk Partner
Port Priority Lacp    A/I Timeout Key   Key   AggrId Id    Port    Status
---- -------- ------- --- ------- ----- ----- ------ ----- ------- ------
23   32768    Active  A   Short   10    8202  8193   6     23      Active
24   32768    Active  A   Short   10    8202  8193   6     24      Active
460-24T-PWR(config)#show spanning-tree port 23,24
Port Trunk   Participation   Priority  Path Cost    State
---- -----  ---------------  --------  ---------  ----------
23   6      Disabled
24   6      Disabled

I disconnected on the links and observed a single ping loss between the two switches. I then reconnected them and everything recovered as expected.

Is this switch a 24 port Ethernet Switch 470?
Are you using the overlapping GBIC/SFP ports which I believe are 25 and 26?

I'm wondering if that's causing you an issue. Are you removing the GBIC/SFP?

Anything of interest in the logs?

Good Luck!

[pw]

Hey Michael,

thanks for taking the time to rebuild my setup!
Interesting, that you are unable to reproduce the problem.

The Switch I'm using is a 24 + 2 Port Switch but Port 25,26 are
not overlapping.
I'm not removing the GBIC but only deactivating it on the connected Switch.

I spend some time wandering around in the shell/menu but could not find much.

p - STP Menu
   
     e-     STP MLT Info
          STPG number? (1-8): 1
          My module number 0
          Module MLT info: pcfgBrdg 0x0 0xfffffe, pcfgBrdgngMlt 0x0 0x0, pcfgPartition 0x0 0x0.
          stp_mltConnErrMask 0x0 0x0. pcfgFastLearning 0x0 0x0

remove and reconnect one fiber (LACP Uplink)
     
      e-     STP MLT Info
        STPG number? (1-8): 1
        My module number 0
        Trunk 6: desgPort 0x10019, portCnt 2, portMask 0x0 0x3000000, activeMask 0x0 0x3000000, rcvMask 0x0 0x0.
        Module MLT info: pcfgBrdg 0x0 0x3fffffe, pcfgBrdgngMlt 0x0 0x3000000, pcfgPartition 0x0 0x0.
        stp_mltConnErrMask 0x0 0x0. pcfgFastLearning 0x0 0x0
        MLT port 25
        MLT port 26
        StpMask (T6):0x1
        Unit 0, port 25
        Unit 0, port 26

Thanks
Philipp

[Michael McNamara]

That's right the GBIC ports are not overlapping on the 24 port model, only on the 48 port model.

That's really odd then... the Juniper is running normal STP and not RSTP or MSTP?

[Artur Gmaj]

Hi Gents,
I have not been active on this forum for quite a lot of time due to my bussines activities. Quite interesting issue. I saw You were asking about static LAG on Juniper. It's possible. Assumig ge-0/0/0 and ge-0/0/1 form static LAG config should be like this:

set chassis aggregated-devices ethernet device-count 1
delete interface ge-0/0/0 unit 0
delete interface ge-0/0/1 unit 0
set interface ge-0/0/0 ethernet-options 802.3ad ae0
set interface ge-0/0/1 ethernet-options 802.3ad ae0
set interface ae0 unit 0 family ethernet-switching

You can find more informathion there:
http://www.juniper.net/techpubs/en_US/junos11.4/topics/concept/interfaces-lag-overview.html

Cheers,

Artur