In my earlier hands on lab post ( Hands on Lab – Replacing Failed Disks from ZFS Pools ( Simple / Mirrored / RaidZ )). I have explained the disk failure scenarios for ZPOOL mirror and RaidZ volumes, and this post is just continuation to that post. And in this post I will showing the multiple disk failure scenarios for the RAIDZ2 and RAIDZ3, and the way to recover it.
In one of our previous post “Solaris Troubleshooting (Magic of Solaris 10) – Root Password Recovery for any Solaris 10 (without CD/DVD)”. We tried to show how to recover system’s root password in Failsafe mode without any media for OS having single disk (i.e native device, c#t#d#s#). But if your system is mirrored then you wont be able to recover the same without loading SVM module in failsafe (also with any other media) mode. As your box wont allow you to mount the metadevice. In this post I will try to present the procedure to load SVM module in such cases.
We are living in a trend of remote System Administration and in real-time we rarely touch and feel a server that we are managing for 24×7 ( 24 Hours a Day and 7 days a Week). Although we don’t really deal with our servers physically, it is always better to know about the hardware that we have to support remotely. The Physical Server hardware awareness helps us for quick decision-making during the hardware troubleshooting.
A decade ago, when I was learning my system administration course, the most challenging task for me is imagine about how a real-time servers looks like and how those servers are different from regular PCs. I couldn’t touch and feel a real server until I joined to my first job as UNIX admin and the server is Enterprise 450 ( a real elephant in shape and weight).
Today, Sysadmin learners are lucky enough to watch the real-time servers before they actually work on it, and the courtesy and credit goes to Oracle. Below are some of the important and powerful servers from oracle which are available in market for mission critical UNIX environment.
I came across one unique parameter which I would like to share on our gurkulindia, specially for small business organisations where we dont have seperate power backups for DC.
Has anyone noticed while working on Solaris boxes if you will power down (init 5 or poweroff from LOM/ALOM/ILOM) the Solaris boxes, they wont come up automatically after physical power on (only LOM boots but No OS). Which is a serious concern if any power outage occur or any power unit failure occurs etc etc…
Ex: Suppose a power outage occurred in any DC and 100 Solaris boxes went down. Now after power restoration, you wont be able to see any Solaris Box up and running. Manual intervention is required and SA need to power on the boxes manually from LOM prompt. Practically this situation is highly unacceptable in any organisation and this is the default feature of Solaris.
After a long investigation and googling and PDF’s I safely landed on earth 
and the landmark is below. There is a parameter on LOM/ALOM (for Sparc arch.) and ILOM (for intel arch.) which you wont find in help. The default value for that parameter is FALSE which wont allow your OS instance to boot up and hence it just stuck and wait for manual intervention at LOM level only.
I have tested the same in large scaled Solaris environment while performing hardware replacements & upgrades, when the value is FALSE the server stuck at LOM and OS wont come up, on the other hand if the value is changed to TRUE and power restored the Box comes up. Full Story
System Controller battery (BATTERY at SC/BAT/V_BAT has exceeded low warning) failure in T-series servers.
Many a times you will get a hardware error in /var/adm/messages (or at ALOM/LOM prompt, console messages) for SC battery failure, In this post I will explain the battery replacement procedure along with its importance in the system.
sc> showfaults -v
Last POST run: THU AUG 09 11:08:20 2012
POST status: Passed all devices
ID Time FRU Fault
0 AUG 09 11:05:54 SC/BAT BATTERY at SC/BAT/V_BAT has exceeded low warning threshold.
Importance:
Battery in T-series model is located on System Controller (SC) and system downtime is required to replace the battery.
The system controller contains the persistent storage for the host ID and Ethernet MAC addresses of the system, as well as the ALOM CMT configuration including the IP addresses and ALOM CMT user accounts, if configured.
Battery provides power to the System Controller NVRAM when no AC power is applied to the system.
Note:
After SC battery replacement the date/time on the SC/host will set to default value as shown below, the same will reflect in the hosts UPTIME too if you wont set the date/time correctly after battery replacement:
sc> showdate
SAT JAN 01 00:06:55 UTC 2000 Full Story
About Author : My name is Ramkumar Ramadevu ( Ramdev ). I have started writing about enterprise unix system administration since 2009 just for my own knowledge reference, and then later I have made this site available for everyone, for better purpose. ... read more