ESXi Version: 5.0.0
Build: 623860
Server: DL 360 G7
SAN Switches: Cisco small business switches
SAN: HDS AMS 2100
Hi, I recently configured software iSCSI to connect to datastores on my SAN. I followed the best practises guide on the vmware website to setup software based iscsi conenction to datastores with multipath and the default policy set was Round Robin.
At roughly 0815 for the past two mornings all connections to the datastore drop off causing the VMs to go offline. The host itself remains online. Around 10 minutes later the connections restore and I am able to power on the VMs and continue as normal.
Here are some of the errors I receive around the time of dropped connection.
syslog.log
2013-01-28T08:06:08Z iscsid: cannot make a connection to 192.168.1.100:3260 (101,Network is unreachable)
2013-01-28T08:06:08Z iscsid: session login failed with error 4,retryCount=0
2013-01-28T08:06:08Z iscsid: Login Target Failed: iqn.1994-04.jp.comysan.t.53.00a6 if=iscsi_vmk@vmk2 addr=192.168.1.100:3260 (TPGT:1 ISID:0x2) err=4
2013-01-28T08:06:08Z iscsid: Login Failed: iqn.1994-04.jp.co.comysans.t.53.00a6 if=iscsi_vmk@vmk2 addr=192.168.1.100:3260 (TPGT:1 ISID:0x2) Reason: 00040000 (Initiator Connection Failure)
2013-01-28T08:16:53Z iscsid: DISCOVERY: transport_name=iscsi_vmk Pending=0 Failed=4
2013-01-28T08:16:54Z iscsid: DISCOVERY: transport_name=bnx2i-3cd92bfb2994 Pending=0 Failed=0
2013-01-28T08:16:54Z iscsid: DISCOVERY: transport_name=bnx2i-3cd92bfb2996 Pending=0 Failed=0
2013-01-28T08:16:55Z iscsid: DISCOVERY: transport_name=bnx2i-68b599c8e30c Pending=0 Failed=0
2013-01-28T08:16:55Z iscsid: DISCOVERY: transport_name=bnx2i-68b599c8e30e Pending=0 Failed=0
2013-01-28T08:16:55Z iscsid: DISCOVERY: transport_name=bnx2i-68b599c8d3f8 Pending=0 Failed=0
2013-01-28T08:16:55Z iscsid: DISCOVERY: transport_name=bnx2i-68b599c8d3fa Pending=0 Failed=0
2013-01-28T08:37:41Z iscsid: cannot make connection to 192.168.2.101:3260 (101)
2013-01-28T08:37:41Z iscsid: connection to discovery address 192.168.2.101 failed
2013-01-28T08:37:41Z iscsid: connection login retries (reopen_max) 5 exceeded
2013-01-28T08:37:41Z iscsid: discovery_sendtargets::Running discovery on IFACE iscsi_vmk@vmk2(iscsi_vmk) (drec.transport=iscsi_vmk)
2013-01-28T08:37:41Z iscsid: discovery_sendtargets::Running discovery on IFACE default(iscsi_vmk) (drec.transport=iscsi_vmk)
2013-01-28T08:37:41Z iscsid: discovery_sendtargets::Running discovery on IFACE iscsi_vmk@vmk1(iscsi_vmk) (drec.transport=iscsi_vmk)
2013-01-28T08:37:41Z iscsid: discovery_sendtargets::Running discovery on IFACE iscsi_vmk@vmk2(iscsi_vmk) (drec.transport=iscsi_vmk)
vmkwarning.log
2013-01-28T08:06:10.174Z cpu23:4739)WARNING: ScsiDeviceIO: 6235: The device naa.60000007e0 does not permit the system to change the sitpua bit to 1.
0:00:00:03.683 cpu0:4096)WARNING: CacheSched: 801: Already disabled : Cache aware scheduling already disabled
0:00:00:03.693 cpu0:4096)WARNING: SVGAConsole: 266: Extended TTY not supported. Ignoring on tty 4
2013-01-28T08:15:06.946Z cpu12:4739)WARNING: Tcpip_Vmk: 716: Failed to set default gateway (51): Network unreachable
2013-01-28T08:15:06.950Z cpu12:4739)WARNING: Tcpip_Vmk: 716: Failed to set default gateway (51): Network unreachable
2013-01-28T08:15:07.091Z cpu12:4739)WARNING: iscsi_vmk: iscsivmk_ModuleInit:501:Offloading digest calculation using CRC32 instruction set
2013-01-28T08:15:10.366Z cpu22:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhba38:CH:0 T:0 CN:0: iSCSI connection is being marked "ONLINE"
2013-01-28T08:15:10.366Z cpu22:4798)WARNING: iscsi_vmk: iscsivmk_StartConnection: Sess [ISID: 00023d000001 TARGET: iqn.1994-04.jp.co.hitachi:rsd.d8s.t.53166.0e0a6 TPGT: 1 TSIH: 0]
I've checked all my VMs and there are no high network io tasks running at any time in the morning, and there are no backup operations configured around the time of failure either.
I was planning to change the Multipath policy tonight to see if this helps at all.
VMware technical support think its a problem with physical switching issue, they asked me to make a note of the mac address now and when the issue arises again to see if they match up. I really dont want to wait around for this to happen for a third time. Any help will be much appreciated
Thanks