I'm not so sure VMWare supports the load-balance configuration all of our Cisco switches and teamed servers utilize.
Thie command sh etherchannel load-balance shows we are in src-mac:
From some reading it seems like VMWare only supports src-dst-ip. I am not going to risk breaking tons of port channels with other equipment or switches to make this change.
Perhaps I should just leave everything as-is since its working. Though I would of thought I could have better bandwidth with link aggregation for at least SAN traffic.
The 3 unconnected ports on each ESX host would just go to the new switch and in VMWare I would just attach them to each vSwitch (noting proper vlan).
vSwitch 1 - SAN (vmnic4)
vSwitch 2 - DMZ (vmnic3)
vSwitch 3 - LAN (vmnic1)
Seems like VMWare ESX 4.1 is just too limited for what I wanted to accomplish. I would love to be told otherwise.