Network Design Issues

[bylie]

Hi,

Several years ago my colleague and I inherited an approx. 1000+ users LAN which was build along the concepts of a collapsed backbone. At the heart of the network are 2 clustered Nortel ERS 8600's which do all of the inter VLAN routing. At the second or distribution layer are Nortel ERS 4500GT's which are uplinked to the backbone via 2 SLT's. Finally at the third or access layer there is a mix of Nortel ERS 2500T's, Nortel ES 470's and Nortel ES 325's to provide connectivity to the endusers.

Because this is a production network for quite a lot of users we were very cautious in the beginning to make changes to the existing configuration. The problem was that we were constantly trying to figure out why some things were as they were and when new networkchanges had to be done we were trying to fit them into the existing configuration as best as possible. As you can imagine this doesn't really scale well , it often took more time to decide where to fit in new equipment logically than to actually install it physically.

So we've now decided to gradually change the existing configuration according to our own IP and VLAN plan. But before we go out and change a lot of stuff we've got some unanswered questions:

VLAN Sizing
In the current network each edgeswitch has it's own VLAN kinda mapped to a physical location like a classroom of office. while we agree that this is probably a very safe and conservative approach it gives us quite some management overhead. We were thinking of moving to a 5 x 24port switches per VLAN approach giving us approx. 120 accessports in each VLAN which is half of what is considered (here and there on the internet) the maximum amount of useable (windows)hosts in a VLAN considering broadcasts and /24 subnets. What are you guys doing and what would you recommend us here?

Nortel ERS 2500T
We've got quite a lot of Nortel ERS 2500T's at the edge of our network and while they're not really letting us down we sometimes get the gutfeeling that they're not all that on the performanceside of things. These are of course only 100Mb/s switches with 1Gb/s uplinks but sometimes our helpdeskguys complain about them when they're doing intensive stuff like imaging a classroom over the network.
When I was troubleshooting one of these a couple of days ago via JDM and using the graph option I saw that under the "Misc" tab quite some ports had, on average, a couple of thousand NoResourcesPktsDropped. If I remember correctly this value is a good indicator of performanceissues related to memorybuffers. When I checked this value on some other 2500's they all had this. Could it be that these switches are not cutting it or is this not as problematic as I'm thinking? Our Nortel ERS 4500GT's for example show all 0's for the NoResourcesPktsDropped value. I'd really like to know what experience others have with these as we're thinking of further replacing all of the aging Nortel ES 470's and 325's in the edge with Nortel ERS 2500T's.

I hope this all makes sense and that you guys can give some well needed best practices and advice!

[Michael McNamara]

I would suggest you have a look at the Large Campus Technical Configuration Guide for starts.

The design that you have doesn't sound overly complex. Hopefully every switch/closet is home run back to the ERS4500s.

We currently use a VLAN per closet with a /23 IP network. We have closets which have 8 switches in a stack so that's 384 ports although recently we've gone beyond a single stack of switches to two stacks of switches quickly approaching the limits of our IP addressing scheme. Our closets are strategically located throughout our facilities so there is no location more than 300' from a closet. On some of our campuses this leaves us with over 46 closets and 46+ VLANs to manage. We use just edge and core with no distribution. The 8600 switches are fiber hotels with SLT links to each edge switch. We have a few high traffic closets where we deploy multiple uplinks in a MLT/SMLT to provide adequate bandwidth.

You could certainly change your VLAN boundaries if you feel they are overly complicated or cumbersome. You could use the building wing or floor or some other geographic boundary. You can also increase your IP network size. We use to have /20 IP networks which produced a lot of background noise in such a large broadcast domain. I would recommend that you don't use anything larger than a /22 given how chatty Microsoft Windows can be. We also rely on our 8600 switches to perform all the routing between the VLANs and WAN.

The ERS2500 is a great switch for the price point. It's meant to compete against the Cisco 2960 so you need to keep that in mind. With that said the switch has been tested at line rate with 64Kb frames and it should be able to keep up with you, although as your frame sizes increase you could run into some issues if your network is really heavily loaded.

I might have missed a few of your questions so please feel free to follow-up with additional questions.

Good Luck

[bylie]

I've actually read the "Large Campus Technical Configuration Guide" and while it gives good best practices on topics like SLPP, STP, (S)MLT's, ... I can't really recall anything on best practices for VLAN sizes and such. It's probably also not that easy to recommend something that will work for everyone in every situation. The only information you find about this on the internet is some (old) Cisco statements that everyone always seems to quote, my turn :

Maximum number of hosts per VLAN, which are a little simplistic:
   
   - 1000 for "well behaved IP" -- hosts that just do ARPs, DHCP, etc. but no broadcast applications
   - 500 for general IP with rational broadcast/multicast applications
   - 200-500 for Apple, DECnet, CLNP -- assuming that hosts on the network will ignore multicasts not meant for them
   - 200 for NetBEUI/windows clients

These numbers reflect only the broadcast/multicast interrupt load on older processors connected at 10 Mbps.

What makes it even more difficult is that some our users have their own ideas about how the network should be configured (we're an educational environment ) and some of them are a bit weary of going from the old one VLAN per classroom to a 5 24p switches per VLAN model because of all the "evil" broadcasts and such. We've actually been running the 5 24p switches per VLAN model on a handfull of locations already as sort of a test and haven't experienced any degradation. But it's sometimes hard to convince them that it won't be "the end of the world", the fact that some of them have a couple of homegrown servers in these client VLAN's probably also won't help as they're clinging on to the old subnets .

So if I understand it correctly you've got locations where 384 active ports/users reside in the same VLAN and don't see any detrimental effects on networkperformance? So we're thinking even more conservatively here by going with the half of a C-class VLAN size.

You're correct about the edgeswitches, every edgeswitch is uplinked to an ERS 4500GT which actually resides in the wiring closet. So in general we try to put an ERS 4500GT in every wiring closet which has an MLT to the backbone and in turn provides Gb/s connectivity to the underlying stacks or classrooomswitches.

We're convinced that for the price the ERS 2500T is a good switch but we're concerned about the NoResourcesPktsDropped value indicating some sort of memorybuffer issue, it always stuck with me that this is a good indicator about the switch's performance not cutting it. One of the techsupport guys from an external firm once said that in our SAN environment we especially had to watch out for this value indicating that the SAN switches (ERS 5510's) were having a hard time. We actually configured our SAN switches to maximize memorybuffer usage via the following commands:

Code: [Select]

(config)# qos agent buffer maximum
(config)# qos agent queue-set 1

[bwilliams2]

We use a /18 at our campuses. And we typically have three VLANs per campus. One data, one voice, and one LWAPP. I don't have any experience with 2500 switches.

[Michael McNamara]

We use a /18 at our campuses. And we typically have three VLANs per campus. One data, one voice, and one LWAPP. I don't have any experience with 2500 switches.

Unless my math is wrong that's 64 Class C networks...  I can understand folks wanting to keep the design simple and minimize the networking staff but I can't imagine the amount of background noise in one of those VLANs. Have you ever performed a baseline packet/sniffer trace? How much of the traffic was needless/pointless broadcast?

@bylie

In any case decisions like this can be subjective because it's hard to collect the needed evidence to push an argument ether way. I've personally seen large (noisy/chatty) networks slow big powerful server and impact overall network performance. The issue is finding that fine line where you're not over managing the network, yet you have networks that are reasonably clear of broadcast storms.

In my opinion I would stay between either a /22 or a /23 and see how that might fit with your closet layout. Perhaps you can squeeze a wing or floor into that space, again something that makes sense to you and your team.

Good Luck!h

[bylie]

We use a /18 at our campuses. And we typically have three VLANs per campus. One data, one voice, and one LWAPP. I don't have any experience with 2500 switches.

I'm not really the expert here (duh ) but /18 sized VLAN's, even to me, seem oversized if actually used that way. However the fact that you're using a /18 subnet on a VLAN doesn't necessarily have to mean that there are effectively that many hosts/ports active in that VLAN which might explain why you're not seeing any problems. I can imagine that one could use a 10.0.0.0/8 subnet for a VLAN without problems as long as there is only a reasonable amount of hosts/ports active in that VLAN.

[bylie]

@Michael

Are you using Nortel ERS 2500's or Nortel ERS 4500's yourself because I always get the impression some of you guys are using nothing less than Nortel ERS 5500's which of course is in a totally different league.

[Michael McNamara]

I don't have any 2500s, I have only a few 4500 for very small doctor offices (mostly broadband VPN). The majority of my network is 460, 470, 5510, 5520, 5530, 1648, 8606, 8610.

While the feature sets are worlds apart the 2500 is still a great value (IMO). If you are going to think about replacing the 2500s and you have a tight budget I would suggest you look at the 4500 series. It provides a lot of the features found in the 5500 and is a 10/100/1000 switch (although you already know that

I would suggest you use WireShark to take a baseline of your current network. Just run a wide open trace for about 60 minutes and tabulate the amount of unicast traffic over multicast traffic. The amount of IP over IPX, the amount of AppleTalk, etc. That might help you get an idea of how clean or dirty the network is and help you decide how to proceed.

I think everyone's "experience" helps make an educated decision. I broke up our /20 networks a few years ago because the broadcast traffic was averaging around 12 -15KB/s. You can actually use MRTG or just the ifOutOctets of any switch port to see how much noise there is on any one VLAN.

Good Luck!

[bylie]

We're actually already graphing most of our switches using Cacti. On some of them we've also enabled the graphing of Non-Unicast Packets. While this really helped us out in the past when we had to go looking for broadcaststorms on certain switches I sometimes find it hard to actually interpret these graphs during normal networkusage. For example on our 5 x 24p switches test-VLAN's we're on average getting 10 Non-Unicast Packets/s or about 4kb/s of noise during normal use and this drops to 1 or 2 Non-Unicast Packets/s during non-networkuse (evenings, night, weekends, ...). But I honestly can't say if this is good or bad or something we have to keep an eye on. On our ERS 8600's in the backbone for example we see 100 Non-Unicast Packets/s on average for the uplinks to our VMware ESX farm which probably is due to all the tagged server VLAN's that are squeezed through those uplinks, MS NLB with it's reliance on non-unicast traffic also doesn't help here either I guess .

Anyway I'll give wireshark a go to see what comes out .

[nightwatch]

We never go under /23 in our networks for VLANs. Normally, /26 and /30 for Layer 3 links.

[bwilliams2]

This tread shows how different each network can be. Some people are more "control" orientated and some are more "function" orientated. I am not sure why the subnet size was set to /18 on "my" network. Typically on a large campus of ours you will not see more than 1,300 active nodes on the data VLAN. So from a math stand point there are certainly more numbers available than used. I am one of 50 people in the IT department. I do know that our bandwidth allows for chatty traffic. I believe the network was originally configured by a third party many years ago.

[bylie]

To summarize, we're probably going to go ahead with our initial idea of 5 x 24p switches per VLAN which will give us a max. of 120 active hosts per VLAN. This is half of a class C (/24) subnet which also mitigates DHCP-starvation because we have twice the amount of IP's available per subnet/VLAN.
In a way we will be looking at the network more from a network point of view than classroom/desk/location point of view like it was in the past. I think this will make it easier for us to provision new networkports/VLAN's/subnets in the future because at the end of the day 99% of our clients only care about networkaccess and not about which VLAN/subnet they're in.