This weekend I configured our VMWare infrastructure to connect to two Nortel 8600s via SMLT. Up until now, the VMWare side had 4 active NICs for data traffic, but the 8600s were not configured with any aggregation settings. Things seemed to go fine during the configuration, but I am now seeing lots of duplicate and out of order TCP packets. I'm sure that this has something to do with the SMLT configuration and how it is handling the frames.
Physical setup:
6x Dell R900 servers with 4x 1Gbps data NICs configured as a single vswitch
the data vswitch is configured to trunk 5 VLANs to the Nortel 8600s
2x Nortel 8600s configured with IST. This works, there are 15 switch closets configured as SMLT
Two of the data NICs from each VMHost connects to a blade on each of the 8600s
All of the VLANs are trunked on each of the data NIC ports on the 8600s
-----------
I just want to check to make sure that logically everything is configured properly before I start looking at the code again.
1) Ensure that all VLANs are trunked on switchports
2) Put VMHost in maintenance mode
3) Disable switchports for data NICs
4) Configure MLT on each 8600:
-PortType: Trunk
-VLANS trunked
-Port Members added
-MLTType SMLT (Running Type SMLT)
-Aggregatable Disabled
-VLACP not configured
5) Edit the VMWare VSwitch and ensure that Load Balancing is set to IP Hash
6) Enable switchports for data NICs
7) Exit VMWare Maintenance mode & migrate VMs back onto hosts

Wash rinse and repeat for next VMHost
---------------
Sorry for the length, but I wanted to try and be concise.
-----------------------
Resolved:
For every packet, I was getting four replies - one for each NIC in the link aggregation. On the VMWare side, failure detection was set to 'Beacon Probing'. By changing this to 'Link Status Only' the problem seems to have been resolved.
You can see this setting in a picture the Nortel Documentation, but I must have missed it in the text.