iSCSI and ERS5632FD

[Dominik]

I have the problem that iSCSI doesnīt work on connections across a ERS5632FD SW 6.2.4.
On our ERS5632FD we have connected a couple of  HP Blade Enclosure with VC FlexFabric 10Gb/24-Port Module Firmware 3.30 and a NetApp FAS 2040.

The HP Enclosures are connected with 2 port 10Gig LACP trunks (tagged). The NetApp is connected via a single tagged Link. Nothing really special here.

I can communicate normal with different protocols to the HP Enclosures and NetApp Storage device,
but they can not communicate with iSCSI with each other.

If I connect the 2 (NetApp and HP Enclosure) devices with a crossover cable to each other ,
the iSCSI communication works immediatly.
Only over the ERS5632FD no iSCSI is possible.

I would be very interested if anybody do iSCSI over a ERS5000 switch and if there are any
additional settings to do for iSCSI communication ?

Cheers 

[Telair]

I had something similar once.  In that case it was the devices on the VLAN were all using jumbo frames, but I did not set the switch to use jumbo frames as well.  So it just dropped all the traffic.

[Flintstone]

Hi Dominik,

Have you tried to troubleshoot the issue using a Sniffer as that might point you in the right direction?

CheerZ and goodluck

[Dominik]

I will capture some traces today, maybe I can see in these traces what goes wrong with the
iSCSI communication.

iSCSI is a very commen protocoll in these days, would be very bad if it doesnīt work on ERS5000 switches ?!?

[bylie]

We have an iSCSI backbone running on a stack of two 24 port ERS 5510's. Besides some tweaks regarding the available portbuffer we haven't done anything special to make iSCSI work.

[Dominik]

I have gathered some traces and now I am more confused than before.

I have in my setup 8 VLAN configured on the uplinks to the HP Blade Enclosure.
The tagged 802.1Q trunk is working normal and I can communicate normal over a routed uplink to all
Interfaces of the HP Blade Enclosure.

The Netapp is also connected via a tagged Uplink with 8 VLANs.
I can communicate via a routed uplink to both devices.
The HP Enclosure and the NetApp can not communicate to each other in the iSCSI VLAN.
This is very strange in fact they are direct connected via L2 in the same VLAN to each other.
I can see in the traces that ARP request in the iSCSI VLAN are sended from HP Enclosure to the NetAPP,
and I can also see taht the NetApp send back the correct ARP repley.
The HP Enclosure sends 5x 2 ARP Broadcasts and stops afterwards.

Not sure why here no comminiction is possible.

Any ideas are welcome.

[Flintstone]

Hi Dominik,
A very strange issue indeed 

It looks like your HP enclosure is not seeing the ARP replies for some reason.  I presume the ARP replies are in the correct Vlan with correct tagging etc?

CheerZ

[Michael McNamara]

Hi Dominik,

Are the ARP frames tagged with the proper VLAN ID (802.1q header)?

This is a single ERS5632?

[Dominik]

The VLAN IDs are all correct, I checked that sveral times.
It is only a single ERS5632FD, a very simple straight forward network design with only one switch.

I reconfigured the 2 Uplinks to the HP Enclosure and made two untagged single Uplinks instead the tagged LACP trunk.

Same behaiviour with these 2 untagged connections, on one connection I have the Data VLAN,
here I am able to communicate normal to the HP Enclosure. On the second Link I added the iSCSI VLAN,
here a communication direct to the Enclosure via the L3 gateway is no problem, a communication from the Enclosure to the NetApp Storage device connected to the same VLAN fails.

Not sure but I think the switch is doing its jop, see no packet drops or errors.
Maybee here is the problem on the NetApp or on the HP VC FlexFabric 10Gb/24-Port Module.

[Flintstone]

Hi Dominik,

Check out Packetlife.net as Stretch discusses using layer 2 switches with iSCSI:

http://packetlife.net/blog/2012/jan/16/catalyst-2960s-iscsi-optimization/

CheerZ

[Jon Hurtt]

This document might help...

Data Center iSCSI Solution Guide
http://support.avaya.com/css/P8/documents/100123892

[Dominik]

THX @Jon and Flinstone

Very valuable informations about iSCSI in the two sources.

At the moment it looks like as the ERS5632 is doing his job and forward packets.
I think the problem is located at the NetApp or on the HP VC FlexFabric 10Gb/24-Port Module.

Not sure wich of these two doesnīt work but I will investigate that.

Cheers

[Michael McNamara]

That's the problem with all these fabric interconnects... they are very difficult to troubleshoot.

I'd be interested in hearing your thoughts and experiences with HP VC FlexFabric 10Gb/24-Port Module if you have time over in the HP Networking. I'm building another data center and I'm probably going to say with the original HP VC Flex10 and VC SAN Interconnects.

[Dominik]

After a lot of testing I was able to get iSCSI running over the ERS5632.

We update the NetApp from a 7.x to 8.x SW release.
Since this update it is possible to get an iSCSI share on our HP enclosures.

The uplink on the NetApp has to be an access port, if we activate tagging on this uplink no communication to the iSCSI shares are possible.
It looks like as if NetApp dosnīt have implemented 802.1Q tagging RFC conform.
So the workaround is to use accessports for the NetApp uplinks, wich is not the perfect solution.

Cheers