It is often necessary to system administrators to determine the multipath status, when ever they receive any alert / error / warning related to the storage. And in my previous post I have discussed about idenification of multipath product in solaris server, In this post i will be talking about the diagnosing multipath status using the different multipath commands based on the multipath product we are using. This is Will help you to quickly identify and escalate the storage path related issues to the storage team, and also helps us to narrow down the issue to faulty HBA / Cable .

EMC POWERPATH

Determining General Path Status:

root# /etc/powermt display dev=all
Pseudo name=emcpower0a
Symmetrix ID=000190103266
Logical device ID=0129
state=alive; policy=SymmOpt; priority=0; queued-IOs=0
==============================================================================
—————- Host —————   – Stor -   — I/O Path -  — Stats —
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
3075 pci@780/pci@0/pci@8/emlx@0,1/fp@0,0 c0t50060482D52FB8A9d65s0 FA 10aB   active  alive    0      0
3077 pci@7c0/pci@0/pci@8/emlx@0/fp@0,0 c3t50060482D52FB886d65s0 FA  7aA   active  alive      0      0
Pseudo name=emcpower0a
CLARiiON ID=APM00041500888 [solaris-server]
Logical device ID=60060160D63616002C161DB37939DE11 [6701-668-4-69-Boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP A, current=SP A       Array failover mode: 1
==============================================================================
—————- Host —————   – Stor -   — I/O Path -  — Stats —
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
3073 pci@780/pci@0/pci@8/emlx@0,1/fp@0,0 c0t50060163006005B9d0s0 SP A3  active  alive      0      0
3073 pci@780/pci@0/pci@8/emlx@0,1/fp@0,0 c0t5006016B006005B9d0s0 SP B3  active  alive      0      0
3077 pci@7c0/pci@0/pci@8/emlx@0/fp@0,0 c3t50060162006005B9d0s0 SP A2    active  alive      0      0
3077 pci@7c0/pci@0/pci@8/emlx@0/fp@0,0 c3t5006016A006005B9d0s0 SP B2    active  alive      0      0

Veritas DMP

Determining DMP Version

DMP is tightly integrated with Veritas Volume Manager and is effectively considered a feature of Volume Manager. The version of DMP can be determined a couple of ways (including pkginfo). However, the recommended method is to check the loaded kernel module

Syntax:   modinfo -w | grep vxdmp | nawk ‘{print $8}’ | sed -e ‘s/://g’

root # modinfo -w | grep vxdmp | nawk ‘{print $8}’ | sed -e ‘s/://g’

5.0MP3RP1_HF3                                              

root #

Determining Managed LUNs

You can get a list of LUNs managed by VxDMP from vxdmpadm getsubpaths all:

root# vxdmpadm getsubpaths all
NAME         STATE[A]   PATH-TYPE[M] DMPNODENAME  ENCLR-NAME   CTLR   ATTRS
================================================================================
c1t0d0s2     ENABLED(A)   -          c1t0d0s2     disk         c1       -
c1t1d0s2     ENABLED(A)   -          c1t1d0s2     disk         c1       -
c0t50060482D52FB8A9d65s2 ENABLED(A)   -          c0t50060482D52FB8A9d65s2 emc0         c0       -
c3t50060482D52FB886d65s2 ENABLED(A)   -          c0t50060482D52FB8A9d65s2 emc0         c3       -
c0t50060163006005B9d0s2 ENABLED(A) PRIMARY      c0t50060163006005B9d0s2 emc_clariion0 c0       -
c0t5006016B006005B9d0s2 ENABLED    SECONDARY    c0t50060163006005B9d0s2 emc_clariion0 c0       -
c3t50060162006005B9d0s2 ENABLED(A) PRIMARY      c0t50060163006005B9d0s2 emc_clariion0 c3       -
c3t5006016A006005B9d0s2 ENABLED    SECONDARY    c0t50060163006005B9d0s2 emc_clariion0 c3       -
c0t50060163006005B9d1s2 ENABLED    SECONDARY    c0t50060163006005B9d1s2 emc_clariion0 c0       -
c0t5006016B006005B9d1s2 ENABLED(A) PRIMARY      c0t50060163006005B9d1s2 emc_clariion0 c0       -
c3t50060162006005B9d1s2 ENABLED    SECONDARY    c0t50060163006005B9d1s2 emc_clariion0 c3       -
c3t5006016A006005B9d1s2 ENABLED(A) PRIMARY      c0t50060163006005B9d1s2 emc_clariion0 c3       -
c0t50060163006005B9d2s2 ENABLED    SECONDARY    c0t50060163006005B9d2s2 emc_clariion0 c0       -
c0t5006016B006005B9d2s2 ENABLED(A) PRIMARY      c0t50060163006005B9d2s2 emc_clariion0 c0       -
c3t50060162006005B9d2s2 ENABLED    SECONDARY    c0t50060163006005B9d2s2 emc_clariion0 c3       -
c3t5006016A006005B9d2s2 ENABLED(A) PRIMARY      c0t50060163006005B9d2s2 emc_clariion0 c3       -
 
The same command can be used to examine a specific LUN: 
root# vxdmpadm getsubpaths dmpnodename=c0t50060163006005B9d0s2
NAME         STATE[A]   PATH-TYPE[M] CTLR-NAME  ENCLR-TYPE   ENCLR-NAME    ATTRS
================================================================================
c0t50060163006005B9d0s2 ENABLED(A) PRIMARY      c0         EMC_CLARiiON emc_clariion0    -
c0t5006016B006005B9d0s2 ENABLED    SECONDARY    c0         EMC_CLARiiON emc_clariion0    -
c3t50060162006005B9d0s2 ENABLED(A) PRIMARY      c3         EMC_CLARiiON emc_clariion0    -
c3t5006016A006005B9d0s2 ENABLED    SECONDARY    c3         EMC_CLARiiON emc_clariion0    -
 Determining Dead Paths
 Dead paths can also be determined with the “vxdmpadm getsubpaths all”command:
 root# vxdmpadm getsubpaths all
 NAME         STATE[A]   PATH-TYPE[M] DMPNODENAME  ENCLR-NAME   CTLR   ATTRS
================================================================================
 c1t0d0s2     ENABLED(A)   -          c1t0d0s2     disk         c1       -
 c1t1d0s2     ENABLED(A)   -          c1t1d0s2     disk         c1       -
 c0t50060482D52FB8A9d65s2 ENABLED(A)   -          c0t50060482D52FB8A9d65s2 emc0         c0       –                                               
 c3t50060482D52FB886d65s2 DISABLED     -          c0t50060482D52FB8A9d65s2 emc0         c3       –                                               
 c0t50060163006005B9d0s2 ENABLED(A) PRIMARY      c0t50060163006005B9d0s2 emc_clariion0 c0       –                                  

•  c0t5006016B006005B9d0s2 ENABLED    SECONDARY    c0t50060163006005B9d0s2 emc_clariion0 c0       –                                  

 c3t50060162006005B9d0s2 DISABLED   PRIMARY      c0t50060163006005B9d0s2 emc_clariion0 c3       –                                  
 c3t5006016A006005B9d0s2 DISABLED   SECONDARY    c0t50060163006005B9d0s2 emc_clariion0 c3       –                                  
 c0t50060163006005B9d1s2 ENABLED    SECONDARY    c0t50060163006005B9d1s2 emc_clariion0 c0       –                                  

•  c0t5006016B006005B9d1s2 ENABLED(A) PRIMARY      c0t50060163006005B9d1s2 emc_clariion0 c0       –                                  

 c3t50060162006005B9d1s2 DISABLED   SECONDARY    c0t50060163006005B9d1s2 emc_clariion0 c3       –                                  
 c3t5006016A006005B9d1s2 DISABLED   PRIMARY      c0t50060163006005B9d1s2 emc_clariion0 c3       –                                  
 c0t50060163006005B9d2s2 ENABLED    SECONDARY    c0t50060163006005B9d2s2 emc_clariion0 c0       –                                  

• c0t5006016B006005B9d2s2 ENABLED(A) PRIMARY      c0t50060163006005B9d2s2 emc_clariion0 c0       –                                   

 c3t50060162006005B9d2s2 DISABLED   SECONDARY    c0t50060163006005B9d2s2 emc_clariion0 c3       –                                  
 c3t5006016A006005B9d2s2 DISABLED   PRIMARY      c0t50060163006005B9d2s2 emc_clariion0 c3       –                                   
 
In the following example, path checking is done for only one device:
 root# vxdmpadm getdmpnode dmpnodename=c0t50060163006005B9d0s2
 NAME                 STATE        ENCLR-TYPE   PATHS  ENBL  DSBL  ENCLR-NAME
 ==============================================================================
 c0t50060163006005B9d0s2 ENABLED      EMC_CLARiiON 4      2     2     emc_clariion0
 root# vxdmpadm getsubpaths dmpnodename=c0t50060163006005B9d0s2
 NAME         STATE[A]   PATH-TYPE[M] CTLR-NAME  ENCLR-TYPE   ENCLR-NAME    ATTRS
 ================================================================================
 c0t50060163006005B9d0s2 ENABLED(A) PRIMARY      c0         EMC_CLARiiON emc_clariion0    -
 c0t5006016B006005B9d0s2 ENABLED    SECONDARY    c0         EMC_CLARiiON emc_clariion0    -
 c3t50060162006005B9d0s2 DISABLED   PRIMARY      c3         EMC_CLARiiON emc_clariion0    –                                        
 c3t5006016A006005B9d0s2 DISABLED   SECONDARY    c3         EMC_CLARiiON emc_clariion0    –                                         

MPxIO

MPxIO is the native multipathing tool available on Solaris 10 operating systems. However, in some cases, MPxIO may not be enabled in favour of Veritas DMP or EMC PowerPath.

Determining MPxIO Version

The best way of assigning a version to MPxIO is to use the version of scsi_vhci driver, which stands for virtual host controller interface. The version can be found using the following command:

      modinfo | grep scsi_vhci | awk ‘{print substr($NF,1,length($NF)-1)}’

root# modinfo | grep scsi_vhci | awk ‘{print substr($NF,1,length($NF)-1)}’

1.54

root #

Determining Managed LUNs

To determine which LUNs are managed by MPxIO, run the following command:  

Syntax: mpathadm list lu

root# mpathadm list lu   
/dev/rdsk/c6t20000014C3E5E627d0s2
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t20000014C3E5E6ABd0s2     
Total Path Count: 1
Operational Path Count: 1
/dev/rdsk/c6t60060160C8061B00389FD1F40C5CDE11d0s2
Total Path Count: 4
Operational Path Count: 4
/dev/rdsk/c6t60060160758020005D30B1358E4BDE11d0s2
Total Path Count: 4
Operational Path Count: 4
/dev/rdsk/c6t60060160758020005C30B1358E4BDE11d0s2
Total Path Count: 4
Operational Path Count: 4
/dev/rdsk/c6t600601606FDA11005EEECA8C201ADE11d0s2
Total Path Count: 2
Operational Path Count: 2

…

…

…

This effectively tells us that MPxIO is enabled on the host (the fact that you get any output at all) and lists out all the LUNs. Devices listed in this command directly relate to the devices listed in the format command.

However, not all LUNs listed here are LUNs from the SAN. To determine if an individual LUN is connected to the SAN, use the following command (per LUN):

 syntax : mpathadm show lu | grep Vendor

 root # mpathadm show lu /dev/rdsk/c6t20000014C3E5E6ABd0s2 | grep Vendor

      Vendor:  SEAGATE            ß— This indicates a local drive

 root # mpathadm show lu /dev/rdsk/c6t60060160C8061B00389FD1F40C5CDE11d0s2 | grep Vendor

        Vendor:  DGC                ß— This indicates a CLARiiON (CX) LUN

 root # mpathadm show lu /dev/rdsk/c6t60060480000190103266533030304538d0s2 | grep Vendor

        Vendor:  EMC                ß— This indicates a Symmetrix (DMX) LUN

   root #

At some point, it may be necessary to collect both Vendor and Product information as follows:  

syntax: mpathadm show lu | egrep ‘Vendor|Product’

This will provide a more complete picture of what type of arrays are connected to the server.

Determining Dead Paths

It is possible to determine whether there are dead paths simply using the output above (section 6.1.1) and comparing the total path count to the operational path count. If all paths are available, the values will be the same. If the number of operational paths is lower than the number of total paths, then there are paths down:

root # mpathadm list lu
        /dev/rdsk/c6t20000014C3E5E627d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c6t20000014C3E5E6ABd0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c6t60060160C8061B00389FD1F40C5CDE11d0s2
                Total Path Count: 4
                Operational Path Count: 2      
        /dev/rdsk/c6t60060160758020005D30B1358E4BDE11d0s2
                Total Path Count: 4
                Operational Path Count: 2
        /dev/rdsk/c6t60060160758020005C30B1358E4BDE11d0s2
                Total Path Count: 4
                Operational Path Count: 2

However, more detail as to which specific paths are dead can be seen as follows:

  syntax:  mpathadm show lu | nawk ‘/Paths:/,/Target Port Groups/’

The paths should all be ‘OK’ as in the following example:

root # mpathadm show lu /dev/rdsk/c6t600601607D220F006C77BAD6F229DE11d0s2 | nawk ‘/Paths:/,/Target Port Groups/’
        Paths:
                Initiator Port Name:  10000000c975444d
                Target Port Name:  5006016a106027b3
                Override Path:  NA
                Path State:  OK          
                Disabled:  no
                Initiator Port Name:  10000000c975444d
                Target Port Name:  50060162106027b3
                Override Path:  NA
                Path State:  OK          
                Disabled:  no
                Initiator Port Name:  10000000c9754389
                Target Port Name:  50060168106027b3
                Override Path:  NA
                Path State:  OK          
                Disabled:  no
                Initiator Port Name:  10000000c9754389
                Target Port Name:  50060160106027b3
                Override Path:  NA
                Path State:  OK          
                Disabled:  no
         Target Port Groups:
root #

If paths are down, they will be shown as “unavailable” as shown in the example below:

root # mpathadm show lu /dev/rdsk/c6t60060160758020005C30B1358E4BDE11d0s2 | nawk ‘/Paths:/,/Target Port Groups/’
        Paths:
                Initiator Port Name:  10000000c975444d
                Target Port Name:  500601684ba0118c
                Override Path:  NA
                Path State:  unavailable 
                Disabled:  no
 
                Initiator Port Name:  10000000c975444d
                Target Port Name:  500601604ba0118c
                Override Path:  NA
                Path State:  unavailable 
                Disabled:  no
 
                Initiator Port Name:  10000000c9754389
                Target Port Name:  500601614ba0118c
                Override Path:  NA
                Path State:  OK                
                Disabled:  no
 
                Initiator Port Name:  10000000c9754389
                Target Port Name:  500601694ba0118c
                Override Path:  NA
                Path State:  OK                
                Disabled:  no
 
        Target Port Groups:
root #

Determining Controller (SP) Ownership

For active/passive arrays, it is possible to determine which controller currently owns the LUN with the following command:

       syntax: mpathadm show lu | nawk ‘/Target Group Ports/, EOF’

root # mpathadm show lu /dev/rdsk/c6t600601607D220F006B77BAD6F229DE11d0s2 | nawk ‘/Target Port Groups/,EOF’
        Target Port Groups:
                ID:  11
                Explicit Failover:  yes
                Access State:  active                
                Target Ports:
                        Name:  5006016a106027b3
                        Relative ID:  0
 
                        Name:  50060168106027b3
                        Relative ID:  0
 
                ID:  12
                Explicit Failover:  yes
                Access State:  standby         
                Target Ports:
                        Name:  50060162106027b3
                        Relative ID:  0
 
                        Name:  50060160106027b3
                        Relative ID:  0
root #