Advice needed on WAN performance LFN connection

[TheKingSlacker]

Guys,

I have two sites (one each side of Canada) that are connected via 100Mbps MPLS with average latency around 65ms. We been experiencing the poor copy performance related to the windows size of the TCP stacks of Windows 2003 and XP workstations. Users are not noticing the issue as most the their transfers are fairly small. However we have upcoming Exchange migration project where we'll need to copy over around 50GB of PST on various nights during the migration. 

The easy answers is to enable windows scaling and increase the TCP window. However, the Server/Desktop teams doesn't want to do this. So.. The thought is for just this project we setup transfer stations for copying the files but so far we haven't been about to get more that 10Mbps for a single copy stream.

I was thinking another option would be to use UDP based copy tool like TFTP, but can they support more the 2GB files? I've also looked at Commerial application like Aspera's point to point but it's $8100 for a 1 year license. Which we don't have in the budget for the project.

Any recommendations or ideas?

TIA!

The King

[Michael McNamara]

Interesting problem... what other methods of file copying might be more efficient?

You really need a WAN acceleration appliance at both ends to help "fix" the problem permanently.

I'll have to give this one some thought. How big is your Internet pipe at both sites? Is that an option?

[TheKingSlacker]

We love Riverbeds! We have round 90 Riverbeds at my company. We went down the path of looking at placing a Riverbed at the remote side of 100Mbps connection but the properly sized model would be north of $60K. The remote side is a managed Datacenter of an acquisition that we are working on scaling down. So there isn't very much interest at putting a Riverbed at that location. Especially, since in the long term it would be too large if we are re-deploy to another office. Our remote offices at 10Mbps or less MPLS connections.

We've tried using the Riverbed Mobile client but we are not seeing to much of a gain with the PST copying. I'm trying to arrange additional testing.

Our Internet connection collapse to the corporate datacenter which is 200Mbps. The remote datacenter has a 40Mbps connection so I don't see a improvement of using that.

When I did some testing of the connection using IPerf I was able to also saturate connection using UDP protocol. That is why I was looking for a copy tool/utility that would use UDP.

I going do some testing with tuning the TCP stack.. I realized yesterday that I didn't set the GlobalMaxTCPsize. So, even though I enabled window scaling the window size wouldn't go above 65k.

We are getting close to the migration window so I need to find a quick and cheap solution. We're even consider copying the PSTs to Hard Drive and put a tech on Red-eye with it.

any other ideas?

TIA
The King.

[Michael McNamara]

I would certainly investigate changing the TCP window sizes and associated buffers. Wether you change the buffers on the some intermediary desktop/server or on the actual servers themselves is certainly an option. The TCP window size will directly impact the SMB block size if you choose to use Windows file copy (Windows Explorer) functions.

Another option may be the use of NFS over UDP. I beleive this is still supported in RedHat/CentOS. You could copy the PST to the a CentOS Linux server locally and then mount a remote RedHat/CentOS NFS share and copy the data over the WAN. You could copy the data to/from the CentOS Linux servers via Samba (SMB).

It might also be easier if you break up the data into smaller chuncks (use 7zip) and have mulitple servers/desktops copying different pieces of the larger data set at the same time. I was looking around to see if you could use someting like BitTorent in a closed network... but I couldn't really find anything useful.

Good Luck!

[TheKingSlacker]

I figured that I should give an update on this.. I found a great little open source application call UDT4 http://udt.sourceforge.net/

UDT4 uses UDP to perform the copy.. UDT4 is a CLI driven app without the option for wildcard copies. So, for each file need to be copied you need the load the server side application and then execute the transfer command on the receiving side.

For the PST copying that I need to do, I just zipped up files and have the zip file  broken into 5Gb chunks (you're recommendation.) I then perform a MD5 hash, just to validate everything is intact at the receiving end.

So far from my testing, I'm able push around 85 to 90Mbps on 100Mbps circuit with +60ms latency!

TheKing.

[Michael McNamara]

That's some impressive performance gains over the legacy protocols. I noticed that UDT is already being utilized in a number of commercial software products. It's probably only a matter of time before someone wraps the protocol and client/server applications in a nice GUI or web enabled front end application.

I'll be moving around 100TB of data over the next 30 days to a new data center although I'll have a 10Gbps network. I'll be curious to try this out and see if there are any significant performance gains even over a 10Gbps network with < 5ms RTT.

Thanks for posting the follow-up... great information! +1rep

[brian.g]

Hello, I just saw these interesting discussions about using UDT. I would like to post more relevant information, hopefully it can be helpful. (btw, I am the developver of UDT:) )

HSCP ( newbielink:http://sourceforge.net/projects/hscp/ [nonactive]) provides a better user interface for using UDT to transfer data. It integrates UDT into SCP, but when using UDT, encryption is ignored. Using HSCP may not be as fast as using the native UDT due to overhead and configuration issues,  but it should still provide a significant improvement over TCP. I have never used HSCP by myself though and I learned it from some of the UDT users.

Another tool that may be helpful is Sector (sector.sf.net), since you mentioned accessing data in a data center. Sector is a distributed file system and can be setup on clusters. When you access data from Sector, it can be regarded as an advanced version of FTP server as it can aggregate data accross multiple computers with a uniform namespace. The Sector client can download (or upload) data by files and directories, with wildcard supported. Sector also uses UDT to transfer files.

If you have a nice pair of computers (and really really fast disk), you should be able to achieve 9.x Gb/s over 10.

UDT itself is a protocol and a library, which is why it didn't provide a GUI.

[Michael McNamara]

Hi Brian,

Thanks for taking the time to post some additional information regarding UDT and HSCP and obviously for your efforts developing the protocol and sharing it with the general public. It seems like UDT has garnished some attention from a small number of commercial developers. As the protocol becomes recognized hopefully other developers (example; Tim Kosse - FileZilla) will build in support for UDT natively or through the use of libraries and APIs.

I'll definitely check out HSCP, it sounds like a great subject for a blog post after I have some performance numbers.

Cheers!

 

[John T]

@TheKingSlacker that's pretty amazing that you were able to turn that type of performance out of that link.