Saturday, October 25, 2008

Fixing fibre channel connectivity problems on AIX

Unix and storage administration always brings interesting and new problems ready to be solved. Few days ago I've encountered one of those problems. Task was to configure two AIX powered VIO servers to access LUNs on CLARiiON storage, i.e. to make high availability environment with several AIX LPARs accessing storage through VIO servers.

After wiring the environment, i.e. hell of finding which port is right by using only lsslot and instructing technician located in data center 80 km far away, next step was to zone the SAN switches. But something was wrong. Every VIO had one dual port FC HBA and every port was connected to a different SAN switch. But instead of seeing two new WWNs (i.e F-ports) on switch, there were two L-ports.

SW1> switchShow
Area Port Media Speed State Proto
=====================================
...
6 6 id N4 Online L-Port 1 public
7 7 id N4 Online L-Port 1 public
...

Situation on the other switch was OK - connected ports were reported as F-ports. For some reason one port of every HBA was L-port (loop port), and the other was F-port. After trying several things, I've managed to get things working as expected.

Apparently this is quite usual problem with AIX, so here is a cookbook how to fix it. You need to unconfigure FC devices and FC SCSI I/O controller protocol, and then set the link initialization protocol to Arbitrated Loop and Switch (AL).

First, look for FC devices:

# lsdev -H -Ccdriver -F "parent name description"
parent name description
fcs0 fcnet0 Fibre Channel Network Protocol Device
fcs1 fcnet1 Fibre Channel Network Protocol Device
fcs2 fcnet2 Fibre Channel Network Protocol Device
fcs3 fcnet3 Fibre Channel Network Protocol Device
fcs0 fscsi0 FC SCSI I/O Controller Protocol Device
fcs1 fscsi1 FC SCSI I/O Controller Protocol Device
fcs2 fscsi2 FC SCSI I/O Controller Protocol Device
fcs3 fscsi3 FC SCSI I/O Controller Protocol Device
iscsi0 iSCSI Protocol Device
powerpath0 PowerPath Control Device
sissas0 sas0 Controller SAS Protocol
sissas0 sata0 Controller SATA Protocol

From here we can see that FC HBAs are fcsX and FC SCSI I/O protocol devices are fcsciX, where X is in range from 0 to 3. Now, deconfigure devices and set the appropriated link initializaion mode. Don't forget to run cfgmgr at the end.

for i in 0 1 2 3
do
rmdev -Rdl fscsi$i
rmdev -Rl fcs$
chdev -l fcs$i -a init_link=al
cfgmgr -vl fcs$i
done

Now you should see both fabric logins and ports listed as F-ports:

SW1> switchShow
Area Port Media Speed State Proto
=====================================
...
6 6 id N4 Online F-Port 10:00:00:00:c9:76:07:33
7 7 id N4 Online F-Port 10:00:00:00:c9:76:0c:2b
...

0 comments: