During the past two to three years,  many organisations started migrating their Server storage from Old legacy SAN devices ( e.g. EMC Clariion )  to new  powerful SAN storage ( e.g. EMC Symetrix VMAX) because of the low performance and maintenance costs involved with legacy storage.  Depending on the budget allocated for the Storage Migration projects, some organisations  prefer the migration by “direct storage level data replication  using expensive migration tools”,  while the other companies ( who are with limited budget) prefer to do the migrations by host level data replication. In the later method,  the success rate of the migration project directly depends on the skill level and expertise of the  unix administrator who is implementing the migration project.  I believe this hands-on post will give some idea for the solaris admins whoever responsible for the Storage migrations. Before going to this post you might want to refer this post for the pre-planning tasks.

  Full Story

Below instructions useful to deal with a common issue in upgrading an operating system (OS) where the existing partitions, which were fine for the operating system (OS) prior to upgrading, are not large enough to hold the same file systems of the new OS. As a example, one of the partitions which demands larger space constantly is the home directory of users which is typically the slice 7 mounted on directory named “/export/home” . 

 Instructions:

 1. Do a level 0 /usr/sbin/ufsdump of all filesystems that are part of the OS or reside on a disk where you need to resize an OS partition.Example:

 # /usr/sbin/ufsdump 0cfu /dev/rmt/0 /dev/rdsk/c0t3d0s0

NOTE: This is a critical step. If the dump is bad, one will not be able to complete the process, and one will have to restore the system from an older backup before starting over. If possible, do this dump booted CD-ROM and only mount the filesystem you are working on at the time. Full Story

Whenever we detect a new disk in Solaris operating system, the first operation we perform is “Disk label” using the format command. The purpose of this post is to explain the background information of Solaris default disk labeling and also discuss the EFI disk labeling introduced from Solaris  9 and later Versions.

SMI Label:

Default Solaris Disk label , i.e. VTOC ( Volume to Content Table) aka SMI(Sun Microsystems, Inc)  label, contains the information such as logical partitioning, disk geometry and other access characteristics. 

In Sparc Hardware – Disk Label ( or VTOC) resides in the first sector ( 512 bytes) of the disk or logical unit. If we run the format command , first time after a  disk was recognized in the operating system, the format command will look at this first sector and complain about “Missing Label” in case of missing VTOC table. Full Story

As a result of unusual failures such as multiple power failures, Solstice DiskSuite /Solaris  Volume Manager both submirrors of a metadevice mirror may be left in an unusual “Last Erred / Last Erred” state making it impossible to determine exactly which submirror must be fixed/replaced first in order to protect the data stored on the metadevice mirror.

Instructions:

The following are two examples of DiskSuite/Solaris Volume Manager metadevice mirrors with both submirrors indicating that components are in the “Last Erred” state.

Normal DiskSuite/Solaris Volume Manager recovery procedures indicate that the submirror in “Maintenance” state must be fixed BEFORE the submirror in “Last Erred” state. In these examples, it is impossible to determine which submirror must be fixed first to protect data.

The first example below is the metastat output from a simple mirrored metadevice made up of two submirrors built upon single slices of a physical disk, The second example below ist the metastat output from a mirrored metadevice made up of two submirrors comprised of a stripe of three physical components.

Although the metadevices are slightly different, in both cases, the attempted recovery prodcedure is exactly the same. Details of the procedure follow the examples below. Full Story

It is often necessary to system administrators to determine the multipath status, when ever they receive any alert / error / warning related to the storage. And in my previous post I have discussed about idenification of multipath product in solaris server, In this post i will be talking about the diagnosing multipath status using the different multipath commands based on the multipath product we are using. This is Will help you to quickly identify and escalate the storage path related issues to the storage team, and also helps us to narrow down the issue to faulty HBA / Cable .

EMC POWERPATH

Determining General Path Status:

root# /etc/powermt display dev=all
Pseudo name=emcpower0a
Symmetrix ID=000190103266
Logical device ID=0129
state=alive; policy=SymmOpt; priority=0; queued-IOs=0
==============================================================================
—————- Host —————   – Stor -   — I/O Path -  — Stats —
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
3075 pci@780/pci@0/pci@8/emlx@0,1/fp@0,0 c0t50060482D52FB8A9d65s0 FA 10aB   active  alive    0      0
3077 pci@7c0/pci@0/pci@8/emlx@0/fp@0,0 c3t50060482D52FB886d65s0 FA  7aA   active  alive      0      0
Pseudo name=emcpower0a
CLARiiON ID=APM00041500888 [solaris-server]
Logical device ID=60060160D63616002C161DB37939DE11 [6701-668-4-69-Boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP A, current=SP A       Array failover mode: 1
==============================================================================
—————- Host —————   – Stor -   — I/O Path -  — Stats —
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
3073 pci@780/pci@0/pci@8/emlx@0,1/fp@0,0 c0t50060163006005B9d0s0 SP A3  active  alive      0      0
3073 pci@780/pci@0/pci@8/emlx@0,1/fp@0,0 c0t5006016B006005B9d0s0 SP B3  active  alive      0      0
3077 pci@7c0/pci@0/pci@8/emlx@0/fp@0,0 c3t50060162006005B9d0s0 SP A2    active  alive      0      0
3077 pci@7c0/pci@0/pci@8/emlx@0/fp@0,0 c3t5006016A006005B9d0s0 SP B2    active  alive      0      0

  Full Story

You will need the following info handy to proceed with the process:

        1.  ls -l /dev/dsk/c#t#d#s# of the alternate sub-mirror boot device

        2. partition table layout of the failed boot device  

Part I Creating an openboot prom alias to alternate boot device

1. Run metadb -i to confirm the state database replicas on boot device is non-operational. For example:

# metadb -i

flags           first blk            block count
M     p        unknown         unknown         /dev/dsk/c0t0d0s0
M     p        unknown         unknown         /dev/dsk/c0t0d0s0
a      u        16                     1034            /dev/dsk/c0t1d0s6
a      u        16                    1034            /dev/dsk/c1t2d0s3

o – replica active prior to last mddb configuration change
u – replica is up to date
l – locator for this replica was read successfully
c – replica’s location was in /etc/opt/SUNWmd/mddb.cf
p – replica’s location was patched in kernel
m – replica is master, this is replica selected as input
W – replica has device write errors
a – replica is active, commits are occurring to this replica
M – replica had problem with master blocks
D – replica had problem with data blocks
F – replica had format problems
S – replica is too small to hold current data base
R – replica had device read errors

  Full Story