Deployment and understanding of LDOMS (Logical Domains)

Deployment and understanding of LDOMS (Logical Domains):

Ldom technology allows us to allocate a system’s various resources such as memory, CPUs and devices (I/O,Networl interfaces etc) into logical groupings and create multiple, discrete systems each having their own operating system, resources, and identity within a single computer system.

Ex: In General Server Techonoly, an OS is directly interacting with hardware through defined system procedure calls and uses complete hardawre resources. But in LDOM the concept is bit different, a new layer between Hardware and OS has been introduced which is called “HYPERVISOR”. It interacts as a medium between Hardware & OS and hence present the hardware components as defined by users to OS. In this case the resource allocation/representation is be done by HYPERVISOR and hence individual OS can have well defined hardware resources. Also remember in LDOM the allocated devices/resources are called VDEV’s (Virtual Devices).

Note: Ldom techology only supported on T-series servers, sparc-4v architecture with Solaris-10 only.

Role of Hypervisor:
===============

The hypervisor, with its stable sun4v interface, is the centerpiece to creating logical domains. Important points to remember are:

• The hypervisor is the layer between the operating system and hardware.

• The hypervisor implements a stable sun4v interface. The operating system makes calls to the hypervisor, and therefore, does not need to know intimate details about the hardware, even if the platform changes.

• The hypervisor is very thin; it exists only to support the operating system for hardware-specific functions, making it small and simple, which assists in
stability.

• The hypervisor creates a virtual machine allowing the system to be partitioned by exposing some of the resources to a specific partition and hiding others.

• The hypervisor creates communication channels, logical domain channels (LDCs), between domains to provide a conduit for services such as networks and shared devices.

Domain Types:
============

There are several different roles for logical domains, and these are mainly defined by context, their usage defines them. A domain may have one or more of these roles, such as combining the functions of an I/O and service domain:

• Control domain – Creates and manages other logical domains and services by communicating with the hypervisor.

• Service domain – Provides services, such as a virtual network switch or a virtual disk service, to other logical domains.

• I/O domain – Has direct ownership of and direct access to physical input/output devices, such as a PCI Express card or a network device. Can optionally share those devices to other domains by providing services.

• Guest domain – Presents a virtual machine that subscribes to services provided by service domains, and is managed by the control domain.

LDOM Daemons:
==============

There are two LDOM daemons ruuning on the system which are.

• LDMD (Logical Domain Daemon)
• VNTSD (Virtual Network Terminal Server Daemon)

Note: Some common terms used in LDOM are Virtual Machine Description (ldm list-spconfig), Virtual Devices (CPU’s, Memory, I/O Devices), Networking (Virtual network (vnet) device, Virtual network switch (vsw)), Storage (Virtual disk client (vdc) driver, Virtual disk server (vds) driver), Console, Cryptographic Devices etc. I will suggest to refer man ldm for all these terms to have much clear understanding.

In this section, I will expalin and create a Controler Domain (Host Domain or Primary Domain).

Pre-requisites:
============

1.) Solaris OS release level:
=====================

The Solaris OS release level should be at-least Solaris 10 11/06 OS. If not then release upgrade should be required to have LDOM software on your system.

# cat /etc/release
  Solaris 10 11/06 s10s_u3wos_10 SPARC
Copyright 2006 Sun Microsystems, Inc. All Rights Reserved.
  Use is subject to license terms.
       Assembled 14 November 2006

2.) Recommneded Patches:
======================

Its always recommneded to have latest Bundle patch installed on your system. Otherwise make sure below patches should be there in the system. If not present kindly install them before proceeding further:

•118833-36, Kernel update patch
•124921-02, which contains updates to the Logical Domains 1.0 drivers and utilities
•125043-01, which contains updates to the qcn (console) drivers.

Note: Kindly go through the readme file for these patches as they required reconfig reboots.

3.) Firmware Versions:
==================

Its always recommended to have latest firmware level installed on your system. As this ensures the hardware and hypervisor can communicate correctly, and all of the features of the your servers can operate.

We can check our current firmware level at ALOM/OS level as follow:

sc> showhost
Sun-Fire-T2000 System Firmware 6.7.12 2011/07/06 20:03

Host flash versions:
   OBP 4.30.4.d 2011/07/06 14:29
   Hypervisor 1.7.3.c 2010/07/09 15:14
   POST 4.30.4.b 2010/07/09 14:24

# prtconf -V
OBP 4.30.4.d 2011/07/06 14:29

Note: I have carried out my testing on T-2000 server and patched the box to latest firmware level.

4.) Installation of Logical Domains Manager:
===================================

The LDOM package is not a default part of Solaris OS. We have to download it from Oracle site and have to apply that on our system as superuser.

SUNWldm.v – Logical Domains Manager package
SUNWjass – Solaris Security Toolkit packages

I have installed LDOM 1.2 version on my machine as shown below:

# pkginfo -l SUNWldm
   PKGINST: SUNWldm
      NAME: Logical Domains Manager
CATEGORY: application
      ARCH: sparc.sun4v
   VERSION: 1.2,REV=2009.06.25.09.48
   BASEDIR: /
    VENDOR: Sun Microsystems, Inc.
      DESC: LDoms Manager – UltraSPARC CMT virtualization
    PSTAMP: svlpen-on10-020090625094807
INSTDATE: Jun 21 2012 11:44
   HOTLINE: Please contact your local service provider
    STATUS: completely installed
     FILES:       66 installed pathnames
                  11 shared pathnames
                  20 directories
                  18 executables
                6564 blocks used (approx)

# ldm -V

Logical Domain Manager (v 1.2)
Hypervisor control protocol v 1.3
Using Hypervisor MD v 1.1

System PROM:
Hypervisor v. 1.7.3. @(#)Hypervisor 1.7.3.c 2010/07/09 15:1415

OpenBoot v. 4.30.4. @(#)OBP 4.30.4.d 2011/07/06 14:29

Note: This package is installed as a script and not as a package.

Creation of Controller Domain:
=========================

1.) Test the ldom software is working and is communicating with the HYPERVISOR:

# ldm list
——————————————————————————
Notice: the LDom Manager is running in configuration mode. Configuration and
resource information is displayed for the configuration under construction;
not the current active configuration. The configuration being constructed
will only take effect after it is downloaded to the system controller and
the host is reset.
——————————————————————————
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
primary active -t-c– SP 32 32640M 0.1% 5d 4h 54m

Note: a.) Path for ldm comamnd is /opt/SUNWldm/bin/
b.) The “t” in FLAGS shows the domain is not yet up and “n” instead of “t” indicates that the server is up.

2.) Creation of default services, which should always be there in Controller Domains. The services are Disk Services, Console Services and Network services. Below are the steps to create them:

# /opt/SUNWldm/bin/ldm add-vds primary-vds0 primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Any configuration
changes made will only take effect after the machine configuration is
downloaded to the system controller and the host is reset.
—————————————————————————-

# /opt/SUNWldm/bin/ldm add-vcc port-range=5000-5100 primary-vcc0 primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Any configuration
changes made will only take effect after the machine configuration is
downloaded to the system controller and the host is reset.
—————————————————————————-

# /opt/SUNWldm/bin/ldm add-vsw net-dev=e1000g0 primary-vsw0 primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Any configuration
changes made will only take effect after the machine configuration is
downloaded to the system controller and the host is reset.
—————————————————————————-

3.) List the services which we have just created:

# /opt/SUNWldm/bin/ldm list-services primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Configuration and
resource information is displayed for the configuration under construction;
not the current active configuration. The configuration being constructed
will only take effect after it is downloaded to the system controller and
the host is reset.
—————————————————————————-
…
Vds: primary-vds0
Vcc: primary-vcc0
port-range=5000-5100
Vsw: primary-vsw0
mac-addr=0:14:4f:f9:68:d0
net-dev=e1000g0
mode=prog,promisc

4.) Now our next step would be to provide the Resources to controller Domain. We will specify an amount of CPU and memory that should be considered a good starting point. Our Primary Domain resources would be as:

• 1 x MAU (cryptographic) unit – these are bound on a per-core basis, and need
to be set up prior to assigning VCPUs
• 4 x virtual CPUs (1 core on an Ultra SPARC T1 system)
• 1024 Mbytes memory (as we will not be using ZFS to deliver disk services, we
do not need the minimum of 4Gbytes memory.)
• Configuration will be saved as “initial”

# /opt/SUNWldm/bin/ldm set-mau 1 primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Any configuration
changes made will only take effect after the machine configuration is
downloaded to the system controller and the host is reset.
—————————————————————————-
# /opt/SUNWldm/bin/ldm set-vcpu 4 primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Any configuration
changes made will only take effect after the machine configuration is
downloaded to the system controller and the host is reset.
—————————————————————————-
# /opt/SUNWldm/bin/ldm set-memory 1024m primary
—————————————————————————-
Notice: the LDom Manager is running in configuration mode. Any configuration
changes made will only take effect after the machine configuration is
downloaded to the system controller and the host is reset.
—————————————————————————-

5.) Creation and use of newly created configuration:

Now that the control domain is configured the way we want it, we need to store it. The hypervisor will then use this configuration after the next power cycle. First, we will list available configurations and then create a new one called initial, and finally confirm its creation:

# /opt/SUNWldm/bin/ldm list-spconfig
factory-default [current]
# /opt/SUNWldm/bin/ldm add-spconfig initial
# /opt/SUNWldm/bin/ldm list-spconfig
factory-default [current]
initial [next]

Note: We can see the new configuration initial has been created and listed as the configuration to use at the next reboot. If a configuration with the name initial already existed, we would receive an error, and would need to use the ldm remove-spconfig command to remove the existing configuration first.

6.) Now we need to reboot our box for the previous changes which we have made to take effects releasing resources for other logical domains.

# shutdown -i6 -g0 -y

Note: Do not use reboot comamnd. Always use init or shutdown command so that no rc phase gets bypassed.

7.) Enable Daemons, This is the point where we have to enable our vntsd.

# svcadm enable vntsd
# svcs -a | grep -i ldom
# svcs -a | grep -i ldom
disabled       Jun_27   svc:/ldoms/agents:default
online         Jun_27   svc:/ldoms/ldmd:default
online        Jun_27   svc:/ldoms/vntsd:default

8.) Check the services running on the controller domain:

# /opt/SUNWldm/bin/ldm list-bindings primary
———————————————————————
Name: primary
State: active
Flags: transition,control,vio service
OS:
Util: 12%
Uptime: 11m
Vcpu: 4
vid pid util strand
0 0 18% 100%
1 1 13% 100%
2 2 9.8% 100%
3 3 5.4% 100%
Mau: 1
Memory: 124m
real-addr phys-addr size
0×4000000 0×4000000 124m
Vars: reboot-command=boot
IO: pci@780 (bus_a)
pci@7c0 (bus_b)
………………….

Our first controller Domain is ready for use. I will explain and create our Guest Domains in my next post and will cover the complete LDOM stuff in that. Moreover I will try to cover the release upgrade and Ldom patchings for Guest and Controller domains with backout plans.

Reference: Oracle Documentation & support.oracle.com