Background processes in Oracle RAC

Oracle Real Application Clusters (RAC) Specific Background Processes

Oracle RAC is composed of two or more database instances. They are composed of memory structures and background processes same as the single instance database.

Oracle RAC instances are composed of following background processes:
ACMS    — Atomic Control file to Memory Service
GTX0-j  — Global Transaction Process
LMON    — Global Enqueue Service Monitor
LMD     — Global Enqueue Service Daemon
LMS     — Global Cache Service Process
LCK0    — Instance Enqueue Process

DIAG    — Diagnosability Daemon
RMSn    — RAC Management Processes
RSMN    — Remote Slave Monitor
DBRM    — Database Resource Manager (from 11g R2)

PING    — Response Time Agent (from 11g R2)
LMHB   — Global Cache/Enqueue Service Heartbeat Monitor
RCBG  — Result Cache BackGround Process

These processes spawned for supporting the multi-instance coordination.

ACMS (from Oracle 11g)    Atomic Control file Memory Service
In an Oracle RAC environment ACMS background process is an agent that ensures a distributed SGA memory update(ie) SGA updates are globally committed on success or globally aborted in event of a failure.

GTX0-j  (from Oracle 11g)    Global Transaction Process
The process provides transparent support for XA global transactions in a RAC environment. The database auto tunes the number of these processes based on the workload of XA global transactions.

LMON  Global Enqueue Service Monitor    (Lock monitor)

The LMON monitors the entire cluster to manage the global enqueues and the resources and performs global enqueue recovery operations. LMON manages instance and process failures and the associated recovery for the Global Cache Service (GCS) and Global Enqueue Service (GES). In particular, LMON handles the part of recovery associated with global resources. LMON provided services are also known as cluster group services (CGS). Lock monitor manages global locks and resources. It handles the redistribution of instance locks whenever instances are started or shutdown. Lock monitor also recovers instance lock information prior to the instance recovery process. Lock monitor co-ordinates with the Process Monitor (PMON) to recover dead processes that hold instance locks.

LMDx  Global Enqueue Service Daemon

The LMD is the lock agent process that manages enqueue manager service requests for Global Cache Service enqueues to control access to global enqueues and resources. This process manages incoming remote resource requests within each instance. The LMD process also handles deadlock detection and remote enqueue requests. Remote resource requests are the requests originating from another instance. LMDn processes manage instance locks that are used to share resources between instances. LMDn processes also handle deadlock detection and remote lock requests.

LMSx  Global Cache Service Processes

The LMSx are the processes that handle remote Global Cache Service (GCS) messages. Real Application Clusters software provides for up to 10 Global Cache Service Processes. The number of LMSx varies depending on the amount of messaging traffic among nodes in the cluster.

This process maintains statuses of datafiles and each cached block by recording information in a Global Resource Directory(GRD). This process also controls the flow of messages to remote instances and manages global data block access and transmits block images between the buffer caches of different instances. This processing is a part of cache fusion feature.

The LMSx handles the acquisition interrupt and blocking interrupt requests from the remote instances for Global Cache Service resources. For cross-instance consistent read requests, the LMSx will create a consistent read version of the block and send it to the requesting instance. The LMSx also controls the flow of messages to remote instances.

The LMSn processes handle the blocking interrupts from the remote instance for the Global Cache Service resources by:

• Managing the resource requests and cross-instance call operations for the shared resources.
  
• Building a list of invalid lock elements and validating the lock elements during recovery.
  
• Handling the global lock deadlock detection and Monitoring for the lock conversion timeouts.

LCKx  Instance Enqueue process

This process manages the global enqueue requests and the cross-instance broadcast. Workload is automatically shared and balanced when there are multiple Global Cache Service Processes (LMSx). This process is called as instance enqueue process. This process manages non-cache fusion resource requests such as library and row cache requests. The instance locks that are used to share resources between instances are held by the lock processes.

DIAG  Diagnosability Daemon

Monitors the health of the instance and captures the data for instance process failures.

RMSn  RAC Management Service
This process is called as Oracle RAC Management Service/Process. These processes perform manageability tasks for Oracle RAC. Tasks include creation of resources related Oracle RAC when new instances are added to the cluster.

RSMN  Remote Slave Monitor
This process is called as Remote Slave Monitor. This process manages background slave process creation and communication on remote instances. This is a background slave process. This process performs tasks on behalf of a coordinating process running in another instance.

LMHB  Global Cache/Enqueue Service Heartbeat Monitor
LMHB monitors the heartbeat of LMON, LMD, and LMSn processes to ensure they are running normally without blocking or spinning. LMBH trace reports low memory, swap space problem and system average load.

Oracle RAC instances use two processes GES(Global Enqueue Service), GCS(Global Cache Service) that enable cache fusion. The GES and GCS maintain records of the statuses of each datafile and each cached block using global resource directory (GRD). This process is referred to as cache fusion and helps in data integrity.

Oracle RAC is composed of two or more instances. When a block of data is read from datafile by an instance within the cluster and another instance is in need of the same block, it is easy to get the block image from the instance which has the block in its SGA rather than reading from the disk. To enable inter instance communication Oracle RAC makes use of interconnects. The Global Enqueue Service(GES) monitors and Instance enqueue process manages the cache fusion.

Related Oracle Articles: Background processes in normal/stand-alone Oracle Database instance

Wait events in Oracle RAC

Oracle RAC (11g / 12c) wait events

In Oracle 9i and prior, the "gc" was spelled out as "global cache" in the RAC wait events.

In RAC wait event names, CR stands for Consistent Read.
Current - is for DMLs
CR - is for Selects
The difference between CR and Current is that CR requires a block with a given SCN, whereas current does not (current mode fetches the block with the current SCN).

gc buffer busy release =>  specifies the time the remote instance locally spends accessing the requested data block. Caused may be because of not enough memory on your nodes, overloaded interconnect.
gc buffer busy acquire

gc current request 
gc cr request => the time it takes to retrieve the data from the remote cache. Oracle may not pick private interconnect and instead route traffic over slower public network. RAC event similar to buffer busy waits, tune SQL to request less data, tune network latency between RAC nodes, localize data access.

gc cr failure 
gc current block lost => Lost blocks due to Interconnect or CPU. Indicates interconnect issues and contention.
gc cr block lost => Lost blocks due to Interconnect or CPU. Indicates interconnect issues and contention.
gc current block corrupt
gc cr block corrupt

gc current block busy => The current block read request was delayed, most likely an I/O bottleneck. Block is already involved in GC operation, shows hot blocks or congestion.
gc cr block busy => The consistent read request was delayed, most likely an I/O bottleneck. Blocks are busy in another instance, check for block level contention or hot blocks.
gc current block congested => Long run queues and/or paging due to memory deficiency. Current block congestion, check for hot blocks or busy interconnect.
gc cr block congested => Long run queues and/or paging due to memory deficiency. cr block congestion, check for hot blocks or busy interconnect.

gc current block 2-way   => Blocks are busy in another instance, check for block level contention or hot blocks.
gc cr block 2-way => Blocks are busy in another instance, check for block level contention or hot blocks.
gc current block 3-way => Blocks are busy in another instance, check for block level contention or hot blocks.
gc cr block 3-way => Blocks are busy in another instance, check for block level contention or hot blocks.

gc current grant 2-way
gc cr grant 2-way
gc current grant busy
gc current grant congested
gc cr grant congested

gc cr multi block read
gc current multi block request => Full table or index scans.
gc cr multi block request => Full table or index scans.
gc cr block build time
gc current block flush time
gc cr block flush time
gc current block send time
gc cr block send time
gc current block pin time
gc domain validation 
gc current retry
ges inquiry response
gcs log flush sync

Related Oracle Articles:    cluvfy commands RAC     srvctl commands RAC    crs_stat commands RAC

SCAN in Oracle RAC

Single Client Access Name (SCAN) in Oracle RAC

Oracle RAC 11g release 2 introduces the Single Client Access Name (SCAN), which provides a single name for clients to access Oracle Databases running in a cluster and simplify the database connection strings that an Oracle Client uses to connect.

Without SCAN, till Oracle Clusterware 11g release 1, TNSNAMES has 1 entry per node.

RACDB =

  (DESCRIPTION =

    (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 1521))

    (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip)(PORT = 1521))
     . . .

    (ADDRESS = (PROTOCOL = TCP)(HOST = nodeN-vip)(PORT = 1521))

    (CONNECT_DATA = 

     . . . ))

With every cluster change, like node addition or removal, all client TNSNAMES need to be changed.

With SCAN, from Oracle Clusterware 11g release2, only 1 entry per cluster is used, regardless of the number of nodes.

RACDB =

  (DESCRIPTION =

    (ADDRESS = (PROTOCOL = TCP)(HOST = SCANname)(PORT = 1521))

    (CONNECT_DATA = 

     . . . ))

Single Client Access Name (SCAN) - eliminates the need to change TNSNAMES entry when nodes are added to or removed from the Cluster. RAC instances register to SCAN listeners as remote listeners. Oracle recommends assigning 3 addresses to SCAN, which will create 3 SCAN listeners, though the cluster has got dozens of nodes. SCAN is a domain name registered to at least one and up to three IP addresses, either in DNS (Domain Name Service) or GNS (Grid Naming Service). The SCAN must resolve to at least one address on the public network. For high availability and scalability, Oracle recommends configuring the SCAN to resolve to three addresses.

SCAN components in a cluster:

• SCAN Name
• SCAN IPs (3)
• SCAN Listeners (3)

Public network addresses are used to provide services to clients. If clients are connecting to the Single Client Access Name (SCAN) addresses, then we may need to change public and virtual IP addresses as we add or remove nodes from the cluster, but you do not need to update clients with new cluster addresses. SCANs function like a cluster alias. SCANs are resolved on any node in the cluster, so unlike a VIP address for a node, clients connecting to the SCAN no longer require updated VIP addresses as nodes are added to or removed from the cluster. Because the SCAN addresses resolve to the cluster, rather than to a node address in the cluster, nodes can be added to or removed from the cluster without affecting the SCAN address configuration.

Oracle Database 11g release 2 (11.2) instances only register with SCAN listeners as remote listeners. Databases register with SCAN listeners as remote listeners, and also continue to register with all node listeners. It's highly recommended that the clients are Oracle 11g R2 or later version clients, to allow them to fully take advantage of the failover with the SCAN settings.

Because of the Oracle Clusterware installation requirement that we provide a SCAN name during installation, if we resolved at least one IP address using the server /etc/hosts file to bypass the installation requirement, but we do not have the infrastructure required for SCAN, after the installation, we can ignore the SCAN and connect to the databases in the cluster using VIPs. Oracle does not support removing the SCAN address. Oracle recommends that we should not configure SCAN IP addresses in the hosts file. But if we use the hosts file to resolve SCAN name, we can have only one SCAN IP address.

Each cluster will have 3 SCAN-Listeners, combined with a SCAN-VIP defined as cluster resources. The root.sh script automatically configures the Single Client Access Name (SCAN) VIPs and SCAN listeners. When we run root.sh on the first node, all 3 scan IPs are plugged on to the first node’s public interface. The SCAN VIP/LISTENER combination will failover to another node in the cluster, if the current node fails. SCAN VIP addresses must be on the same subnet as virtual IP addresses and public IP addresses.

Cluster Resources
--------------------------------------------

ora.LISTENER_SCAN1.lsnr 1  ONLINE  ONLINE       node1

ora.LISTENER_SCAN2.lsnr 1  ONLINE  ONLINE       node2

ora.LISTENER_SCAN3.lsnr 1  ONLINE  ONLINE       node3

Domain Name Service (DNS)

The SCAN is a fully qualified name (hostname+domain) that is configured to resolve to all the addresses allocated for the SCAN. The addresses resolve using Round Robin DNS either on the DNS server, or within the cluster in a GNS configuration. SCAN listeners can run on any node in the cluster. SCANs provide location independence for the databases, so that client configuration does not have to depend on which nodes run a particular database.

To use the DNS method for defining SCAN, the network administrator must create a single name that resolves to 3 separate IP addresses using round-robin algorithms. Regardless of how many systems are part of cluster, Oracle recommends that 3 IP addresses are configured to allow for failover and load-balancing. SCAN Name needs to be registered in DNS and resolved using round-robin algorithm. To test the round-robin algorithm, perform nslookup on the SCAN Name. The DNS server should return those 3 set of IP Address in different order each time.

It is important that the IP addresses are on the same subnet as the public network for the server. The other two requirements are that the name (not including the domain suffix) are 15 characters or less in length and that the name can be resolved without using the domain suffix. Also, the IP addresses should not be specifically assigned to any of the nodes in the cluster.

Grid Naming Service (GNS)

Grid Naming Service (GNS) is a new feature introduced in Oracle RAC 11g R2.  

As part of the Grid Infrastructure installation process, optionally we can also provide Grid Naming Service (GNS) information, which includes the GNS Sub Domain Name and the GNS VIP Address.

Using GNS assumes that a DHCP server is running on the public network with enough available addresses to assign the required IP addresses and the SCAN VIP. Only one static IP address is required to be configured and it should be in the DNS domain.

Oracle Clusterware 11g release 2 (11.2) supports the use of Dynamic Host Configuration Protocol (DHCP) for the VIP addresses and the SCAN address, but not the public address. DHCP provides dynamic configuration of the host's IP address, but it does not provide an optimal method of producing names that are useful to external clients. When using GNS and DHCP, Oracle Clusterware configures the VIP addresses for the SCAN name that is provided during cluster configuration. The node VIP and the three SCAN VIPs are obtained from the DHCP server when using GNS. If a new server joins the cluster, then Oracle Clusterware dynamically obtains the required VIP address from the DHCP server, updates the cluster resource, and makes the server accessible through GNS.

Summary of points about SCAN:

• SCAN Name will represent the cluster in the network. 
• SCAN used by clients to connect to any database in the cluster.
• SCAN is a GSD resource, which is managed by CRS.
• SCAN provides a single domain name (via DNS), allowing end-users to address a RAC cluster as-if it were a single IP address.
• Removes the requirement to change the client connection if cluster changes.
• Load balances across the instances providing a service. SCAN listener selects least loaded node.
• SCAN requires a DNS entry or GNS to be used.
• SCAN is an alternative to the transparent application failover (TAF) for automatic load balancing.
• Provides failover between instances.
• SCAN VIP/LISTENER will failover to another node in the cluster.
• Clients do not require VIP information.
• Must resolve to at least one address on the public network.
• SCAN provides location independence for the databases.
• Allow clients to use EZConnect or JDBC connections.
• SCAN Listener would forward the request to local listener that’s running on VIPs.
• Each cluster will have 3 SCAN listeners, each having a SCAN VIP defined as cluster resources.
• Instance registers with local listener on its node. Database “REMOTE_LISTENER” registers instances with all SCAN listeners.
• SCAN listeners on each node in the cluster, which are not replacements for a regular listener.
• PMON process of each instance registers the database services with the default listener on the local node and with each SCAN listener, which is specified by the REMOTE_LISTENER database parameter.
• Client queries DNS to resolve to SCAN.

SCAN (Single Client Access Name) commands in Oracle RAC

cluvfy comp scan -> Use the cluvfy comp scan component verification command to check the Single Client Access Name (SCAN) configuration.

crsctl status resource -w "TYPE = ora.scan_listner.type"

SCAN:

srvctl add scan -n scan_name [-k network_number] [-S subnet/netmask[/if1[|if2|...]]]

#srvctl add scan -n scan.mycluster.example.com   -- Adds new SCAN information for a cluster

srvctl remove scan [-f]   -- Removes SCAN information

srvctl remove scan

srvctl remove scan -f

srvctl start scan [-i ordinal_number] [-n node_name]

srvctl start scan

srvctl start scan -i 1 -n node1

srvctl stop scan [-i ordinal_number] [-f]

srvctl stop scan

srvctl stop scan -i 1

srvctl status scan [-i ordinal_number]

srvctl status scan

srvctl status scan -i 1

srvctl enable scan [-i ordinal_number]

srvctl enable scan

srvctl enable scan -i 1

srvctl disable scan [-i ordinal_number]

srvctl disable scan

srvctl disable scan -i 3

srvctl config scan [-i ordinal_number]   -- Shows the current SCAN configuration

srvctl config scan

srvctl config scan -i 2

srvctl modify scan -n scan_name

srvctl modify scan -n scan1
-- Modifies SCAN information (used when changing SCAN to DNS after initially using /etc/hosts)

srvctl relocate scan -i ordinal_number [-n node_name]

srvctl relocate scan -i 2 -n node2

ordinal_number=1,2,3 --> this refers to the SCAN VIPs

Scan_listener:

srvctl add scan_listener [-l lsnr_name_prefix] [-s] [-p "[TCP:]port_list[/IPC:key][/NMP:pipe_name][/TCPS:s_port] [/SDP:port]"]

#srvctl add scan_listener -l myscanlistener   -- Adds a new SCAN listener for a cluster on the default port of 1521

#srvctl add scan_listener -p 1565         -- Adds a new SCAN listener on a different port

srvctl remove scan_listener [-f]

srvctl remove scan_listener        -- Removes the SCAN listener

srvctl remove scan_listener -f

srvctl start scan_listener [-n node_name] [-i ordinal_number]

srvctl start scan_listener

srvctl start scan_listener -i 1

srvctl stop scan_listener [-i ordinal_number] [-f]

srvctl stop scan_listener

srvctl stop scan_listener -i 3

srvctl status scan_listener [-i ordinal_number]

srvctl status scan_listener

srvctl status scan_listener -i 1

srvctl enable scan_listener [-i ordinal_number]

srvctl enable scan_listener

srvctl enable scan_listener -i 2

srvctl disable scan_listener [-i ordinal_number]

srvctl disable scan_listener

srvctl disable scan_listener -i 1

srvctl config scan_listener [-i ordinal_number]  -- Shows the existence and port numbers for the SCAN listeners

srvctl config scan_listener

srvctl config scan_listener -i 3

srvctl modify scan_listener {-p [TCP:]port[/IPC:key][/NMP:pipe_name] [/TCPS:s_port][/SDP:port] | -u }

srvctl modify scan_listener –u -- Modifies the SCAN listener information to match the new SCAN VIP information

srvctl modify scan_listener -p 1521

srvctl relocate scan_listener -i ordinal_number [-n node_name]

srvctl relocate scan_listener -i 1

ordinal_number=1,2,3

GNS (Grid Naming Service):

srvctl add gns -i ip_address -d domain

srvctl add gns -i 192.124.16.96 -d cluster.mycompany.com

srvctl remove gns [-f]

srvctl remove gns

srvctl start gns [-l log_level] [-n node_name]

srvctl start gns

srvctl stop gns [-n node_name [-v] [-f]

srvctl stop gns

srvctl status gns [-n node_name]

srvctl status gns

srvctl enable gns [-n node_name]

srvctl enable gns

srvctl disable gns [-n node_name]

srvctl disable gns -n devnode2

srvctl config gns [-a] [-d] [-k] [-m] [-n node_name] [-p] [-s] [-V] [-q name] [-l] [-v]

srvctl config gns -n lnx03

srvctl modify gns [-i ip_address] [-d domain]

srvctl modify gns -i 192.100.010.007

srvctl relocate gns [-n node_name]

srvctl relocate gns -n node2

Related Oracle Articles:  Virtual IP (VIP) in RAC    srvctl commands RAC    cluvfy commands RAC    crsctl commands RAC

Virtual IP (VIP) in Oracle RAC

What is (use of) Virtual IP (VIP) in Oracle Real Application Clusters (RAC)?

When installing Oracle 10g/11g R1 RAC, three network interfaces(IPs) are required for each node in the RAC cluster, they are:

• Public Interface:  Used for normal network communications to the node
• Private Interface:  Used as the cluster interconnect
• Virtual (Public) Interface: Used for failover and RAC management

When installing Oracle 11g R2 RAC, we need one more network interface(IP) is required for each node in the RAC cluster.

• SCAN Interface (IP):  Single Client Access Name (SCAN) is a new Oracle Real Application Clusters (RAC) 11g Release 2 feature, that provides a single name for clients to access an Oracle Database running in a cluster. The benefit is clients using SCAN do not need to change if you add or remove nodes in the cluster.

When a client connects to a tns-alias, it uses a TCP connection to an IP address, defined in the tnsnames.ora file. When using RAC, we define multiple addresses in our tns-alias, to be able to failover when an IP address, listener or instance is unavailable. TCP timeouts can differ from platform to platform or implementation to implementation. This makes it difficult to predict the failover time. 

Oracle 10g Cluster Ready Services enables databases to use a Virtual IP address to configure the listener ON. This feature is to assure that oracle clients quickly failover when a node fails. In Oracle Database 10g RAC, the use of a virtual IP address to mask the individual IPO addresses of the clustered nodes is required. The virtual IP addresses are used to simplify failover and are automatically managed by CRS.

To create a Virtual IP (VIP) address, the Virtual IP Configuration Assistant (VIPCA) is called from the root.sh script of a RAC install, which then configures the virtual IP addresses for each node specified during the installation process. In order to be able to run VIPCA, there must be unused public IP addresses available for each node that has been configured in the /etc/hosts file.

One public IP address for each node to use for its Virtual IP address for client connections and for connection failover. This IP address is in addition to the operating system managed public host IP address that is already assigned to the node by the operating system. This public Virtual IP must be associated with the same interface name on every node that is a part of the cluster. The IP addresses that are used for all of the nodes that are part of a cluster must be from the same subnet. The host names for the VIP addresses must be registered with the domain name server (DNS). The Virtual IP address should not be in use at the time of the installation because this is a Virtual IP address that Oracle manages internally to the RAC processes. This virtual IP address does not require a separate NIC. The VIPs should be registered in the DNS. The VIP addresses must be on the same subnet as the public host network addresses. Each Virtual IP (VIP) configured requires an unused and resolvable IP address.

Using virtual IP we can save our TCP/IP timeout problem because Oracle notification service (ONS) maintains communication between each nodes and listeners. Once ONS found any listener down or node down, it will notify another nodes and listeners. While new connection is trying to establish connection to failure node or listener, virtual IP of failure node automatically divert to surviving node and session will be establishing in another surviving node. This process doesn't wait for TCP/IP timeout event. Due to this new connection gets faster session establishment to another surviving nodes/listener.

Virtual IP (VIP) is for fast connection establishment in failover dictation. Still we can use physical IP address in Oracle 10g in listener if we have no worry for failover timing. We can change default TCP/IP timeout using operating system utilities/commands and kept smaller. But taking advantage of VIP (Virtual IP address) in Oracle 10g RAC database is advisable.

Oracle RAC SRVCTL commands related to VIP

# srvctl start nodeapps -n node_name   -- Starts GSD, VIP, listener and ONS
# srvctl start nodeapps -n rac99

# srvctl stop nodeapps -n node_name [-r] -- Stops GSD, VIP, listener and ONS
# srvctl stop nodeapps -n lnx03

# srvctl config nodeapps -n node_name [-a] [-g] [-o] [-s] [-l]
    -a                  Display VIP configuration
# srvctl config nodeapps -n node33 -a

# srvctl modify nodeapps -n node_name [-A new_vip_address]
# srvctl modify nodeapps -n lnx06 -A 10.50.99.43/255.255.252.0/eth0

srvctl modify nodeapps [-n node_name -A new_vip_address] [-S subnet/netmask[/if1[|if2|...]] [-m multicast_ip_address] [-p multicast_port_number] [-e eons_listen_port] [-l ons_local_port] [-r ons_remote_port] [-t host[:port][,host:port,...]] [-v]
srvctl modify nodeapps -n node1 -A 100.200.300.40/255.255.255.0/eth0

srvctl add vip -n node_name -A {name|ip}/netmask[/if1[if2|...]] [-k network_number] [-v]
# srvctl add vip -n node96 -A 192.124.16.96/255.255.255.0 -k 2

srvctl remove vip -i "vip_name_list" [-f] [-y] [-v]
srvctl remove vip -i "vip1,vip2,vip3" -f -y -v

srvctl start vip {-n node_name|-i vip_name} [-v]
srvctl start vip -i dev1-vip -v

srvctl stop vip {-n node_name|-i vip_name} [-r] [-v]
srvctl stop vip -n node1 -v

srvctl status vip {-n node_name|-i vip_name}
srvctl status vip -i node1-vip

srvctl enable vip -i vip_name [-v]
srvctl enable vip -i prod_vip -v

srvctl disable vip -i vip_name [-v]
srvctl disable vip -i vip3 -v

srvctl config vip {-n node_name|-i vip_name}
srvctl config vip -n devnode2

srvctl getenv vip -i vip_name [-t "name_list"] [-v]
srvctl getenv vip -i node1-vip

srvctl setenv vip -i vip_name {-t "name=val[,name=val,...]" | -T "name=val"}
srvctl setenv vip -i dev1-vip -t LANG=en

srvctl unsetenv vip -i vip_name -t "name_list" [-v]
srvctl unsetenv vip -i myvip -t CLASSPATH

# olsnodes -i   ==> will show VIPs of the nodes.
node1 168.192.1.1
node2 168.192.2.1
node3 168.192.3.1
node4 168.192.4.1

$ vipca   - VIP Configuration Assistant
$ vipca -nodelist node1,node3

Other Oracle Articles:    crsctl commands RAC      srvctl commands RAC      cluvfy commands RAC    SCAN in RAC

crsctl commands in Oracle RAC

crsctl - Cluster Ready Service Control - commands in 11g Oracle RAC

$crsctl -- to get help

$crsctl query crs activeversion

$crsctl query crs softwareversion [node_name]

#crsctl start crs

#crsctl stop crs

(or)

#/etc/init.d/init.crs start

#/etc/init.d/init.crs stop

#crsctl enable crs

#crsctl disable crs

(or)

#/etc/init.d/init.crs enable

#/etc/init.d/init.crs disable

$crsctl check crs

$crsctl check cluster [-node node_name] -- Oracle RAC 11g command, checks the viability of CSS across nodes

#crsctl start cluster -n HostName -- 11g R2

#crsctl stop cluster -n HostName -- 11g R2

#crsctl stop cluster -all  -- 11g R2

$crsctl check cssd

$crsctl check crsd

$crsctl check evmd

$crsctl check oprocd

$crsctl check ctss

#/etc/init.d/init.cssd stop

#/etc/init.d/init.cssd start

#/etc/rc.d/init.d/init.evmd

#/etc/rc.d/init.d/init.cssd

#/etc/rc.d/init.d/init.crsd

#mv /etc/rc3.d/S96init.cssd /etc/rc3.d/_S96init.cssd -- to stop cssd from autostarting after reboot

#crsctl check css votedisk

#crsctl query css votedisk -- lists the voting disks used by CSS

#crsctl add css votedisk PATH

#crsctl add css votedisk PATH -force -- if Clusterware is not running

#crsctl delete css votedisk PATH

#crsctl delete css votedisk PATH -force -- if Clusterware is not running

#crsctl set css parameter_name value -- set parameters on OCR

#crsctl set css misscount 100

#crsctl unset css parameter_name -- sets CSS parameter to its default

#crsctl unset css misscount

#crsctl get css parameter_name -- gets the value of a CSS parameter

#crsctl get css disktimeout

#crsctl get css misscount

#crsctl get css reboottime

#crsctl start resources -- starts Clusterware resources

./crsctl start resource ora.DATA.dg
#crsctl stop resources -- stops Clusterware resources

$crsctl status resource

$crsctl status resource -t

$crsctl stat resource -t

$crsctl lsmodules crs -- lists CRS modules that can be used for debugging

CRSUI

CRSCOMM

CRSRTI

CRSMAIN

CRSPLACE

CRSAPP

CRSRES

CRSCOMM

CRSOCR

CRSTIMER

CRSEVT

CRSD

CLUCLS

CSSCLNT

COMMCRS

COMMNS

$crsctl lsmodules css -- lists CSS modules that can be used for debugging

CSSD

COMMCRS

COMMNS

$crsctl lsmodules evm -- lists EVM modules that can be used for debugging

EVMD

EVMDMAIN

EVMCOMM

EVMEVT

EVMAPP

EVMAGENT

CRSOCR

CLUCLS

CSSCLNT

COMMCRS

COMMNS

$crsctl start has   (HAS - High Availability Services)
$crsctl stop has
$crsctl check has

OCR Modules -- cannot be listed with crsctl lsmodules command

OCRAPI

OCRCLI

OCRSRV

OCRMAS

OCRMSG

OCRCAC

OCRRAW

OCRUTL

OCROSD

#crsctl debug statedump crs -- dumps state info for crs objects

#crsctl debug statedump css -- dumps state info for css objects

#crsctl debug statedump evm -- dumps state info for evm objects

#crsctl debug log crs [module:level]{,module:level} ...

-- Turns on debugging for CRS

#crsctl debug log crs CRSEVT:5,CRSAPP:5,CRSTIMER:5,CRSRES:5,CRSRTI:1,CRSCOMM:2

#crsctl debug log css [module:level]{,module:level} ...

-- Turns on debugging for CSS

#crsctl debug log css CSSD:1

#crsctl debug log evm [module:level]{,module:level} ...

-- Turns on debugging for EVM

#crsctl debug log evm EVMCOMM:1

#crsctl debug trace crs -- dumps CRS in-memory tracing cache

#crsctl debug trace css -- dumps CSS in-memory tracing cache

#crsctl debug trace evm -- dumps EVM in-memory tracing cache

#crsctl debug log res resource_name:level -- turns on debugging for resources

#crsctl debug log res "ora.lnx04.vip:1"

#crsctl trace all_the_above_commands -- tracing by adding a "trace" argument.

#crsctl trace check css

#crsctl backup -h

#crsctl backup css votedisk

Here is the list of the options for CRSCTL in 11gR2:

       crsctl add       - add a resource, type or other entity

       crsctl backup    - back up voting disk for CSS

       crsctl check     - check a service, resource or other entity

       crsctl config    - output autostart configuration

       crsctl debug     - obtain or modify debug state

       crsctl delete    - delete a resource, type or other entity

       crsctl disable   - disable autostart

       crsctl discover  - discover DHCP server

       crsctl enable    - enable autostart

       crsctl get       - get an entity value

       crsctl getperm   - get entity permissions

       crsctl lsmodules - list debug modules

       crsctl modify    - modify a resource, type or other entity

       crsctl query     - query service state

       crsctl pin       - Pin the nodes in the nodelist

       crsctl relocate  - relocate a resource, server or other entity

       crsctl replace   - replaces the location of voting files

       crsctl release   - release a DHCP lease

       crsctl request   - request a DHCP lease

       crsctl setperm   - set entity permissions

       crsctl set       - set an entity value

       crsctl start     - start a resource, server or other entity

       crsctl status    - get status of a resource or other entity

       crsctl stop      - stop a resource, server or other entity

       crsctl unpin     - unpin the nodes in the nodelist

       crsctl unset     - unset a entity value, restoring its default

crsctl add resource resource_name -type resource_type [-file file_path | -attr "attribute_name=attribute_value,attribute_name=attribute_value,..."] [-i] [-f]
crsctl add resource r1 -type test_type1 -attr "PATH_NAME=/tmp/r1.txt"
crsctl add resource app.appvip -type app.appvip.type -attr "RESTART_ATTEMPTS=2, START_TIMEOUT=100,STOP_TIMEOUT=100,CHECK_INTERVAL=10,USR_ORA_VIP=172.16.0.0, START_DEPENDENCIES=hard(ora.net1.network)pullup(ora.net1.network), STOP_DEPENDENCIES=hard(ora.net1.network)"
crsctl add type type_name -basetype base_type_name {-attr "ATTRIBUTE=attribute_name | -file file_path,TYPE={string | int} [,DEFAULT_VALUE=default_value][,FLAGS=[READONLY][|REQUIRED]]"}
crsctl add type test_type1 -basetype cluster_resource -attr "ATTRIBUTE=FOO,TYPE=integer,DEFAULT_VALUE=0"
crsctl add crs administrator -u user_name [-f]
crsctl add crs administrator -u scott
crsctl add css votedisk path_to_voting_disk [path_to_voting_disk ...] [-purge]
crsctl add css votedisk /stor/grid/ -purge
crsctl add serverpool server_pool_name {-file file_path | -attr "attr_name=attr_value[,attr_name=attr_value[,...]]"} [-i] [-f]
crsctl add serverpool testsp -attr "MAX_SIZE=5"
crsctl add serverpool sp1 -file /tmp/sp1_attr

crsctl check cluster [-all | [-n server_name [...]]
crsctl check cluster -all
crsctl check crs
crsctl check css
crsctl check ctss    -- Cluster Time Synchronization services
crsctl check evm
crsctl check resource {resource_name [...] | -w "filter" } [-n node_name] [-k cardinality_id] [-d degree_id] }
crsctl check resource appsvip

crsctl config crs

crsctl delete crs administrator -u user_name [-f]
crsctl delete crs administrator -u scott
crsctl delete resource resource_name [-i] [-f]
crsctl delete resource myResource
crsctl delete type type_name [-i]
crsctl delete type app.appvip.type
crsctl delete css votedisk voting_disk_GUID [voting_disk_GUID [...]]
crsctl delete css votedisk 61f4273ca8b34fd0bfadc2531605581d
crsctl delete node -n node_name
crsctl delete node -n node06
crsctl delete serverpool server_pool_name [server_pool_name [...]] [-i]
crsctl delete serverpool sp1

crsctl disable crs

crsctl discover dhcp -clientid clientid [-port port]
crsctl discover dhcp -clientid dsmjk252clr-dtmk01-vip

crsctl enable crs

crsctl get hostname
crsctl get clientid dhcp -cluname cluster_name -viptype vip_type [-vip vip_res_name] [-n node_name]
crsctl get clientid dhcp -cluname dsmjk252clr -viptype HOSTVIP -n tmjk01
crsctl get css parameter
crsctl get css disktimeout
crsctl get css ipmiaddr
crsctl get nodename
crsctl getperm resource resource_name [ {-u user_name | -g group_name} ]
crsctl getperm resource app.appvip
crsctl getperm resource app.appvip -u oracle
crsctl getperm resource app.appvip -g dba
crsctl getperm type resource_type [-u user_name] | [-g group_name]
crsctl getperm type app.appvip.type
crsctl getperm serverpool server_pool_name [-u user_name | -g group_name]
crsctl getperm serverpool sp1

crsctl lsmodules {mdns | gpnp | css | crf | crs | ctss | evm | gipc}
crsctl lsmodules evm
mdns: Multicast domain name server 
gpnp: Grid Plug and Play service 
css: Cluster Synchronization Services 
crf: Cluster Health Monitor
crs: Cluster Ready Services
ctss: Cluster Time Synchronization Service 
evm: Event Manager
gipc: Grid Interprocess Communication 

crsctl modify resource resource_name -attr "attribute_name=attribute_value" [-i] [-f] [-delete]
crsctl modify resource appsvip -attr USR_ORA_VIP=10.1.220.17 -i
crsctl modify type type_name -attr "ATTRIBUTE=attribute_name,TYPE={string | int} [,DEFAULT_VALUE=default_value [,FLAGS=[READONLY][| REQUIRED]]" [-i] [-f]]
crsctl modify type myType.type -attr "ATTRIBUTE=FOO,DEFAULT_VALUE=0 ATTRIBUTE=BAR,DEFAULT_VALUE=baz"
crsctl modify serverpool server_pool_name -attr "attr_name=attr_value [,attr_name=attr_value[, ...]]" [-i] [-f]
crsctl modify serverpool sp1 -attr "MAX_SIZE=7"

crsctl pin css -n node_name [ node_name [..]]
crsctl pin css -n node2

crsctl query crs administrator
crsctl query crs activeversion
crsctl query crs releaseversion
crsctl query crs softwareversion node_name
crsctl query css ipmiconfig
crsctl query css ipmidevice
crsctl query css votedisk
crsctl query dns {-servers | -name name [-dnsserver DNS_server_address] [-port port] [-attempts number_of_attempts] [-timeout timeout_in_seconds] [-v]}

crsctl release dhcp -clientid clientid [-port port]
crsctl release dhcp -clientid spmjk662clr-spmjk03-vip

crsctl relocate resource {resource_name | resource_name | -all -s source_server | -w "filter"} [-n destination_server] [-k cid] [-env "env1=val1,env2=val2,..."] [-i] [-f]
crsctl relocate resource myResource1 -s node1 -n node3
crsctl relocate server server_name [...] -c server_pool_name [-i] [-f]
crsctl relocate server node6 node7 -c sp1

crsctl replace discoverystring 'absolute_path[,...]'
crsctl replace discoverystring "/oracle/css1/*,/oracle/css2/*"
crsctl replace votedisk [+asm_disk_group | path_to_voting_disk [...]]
crsctl replace votedisk +diskgroup1
crsctl replace votedisk /mnt/nfs/disk1 /mnt/nfs/disk2

crsctl request dhcp -clientid clientid [-port port]
crsctl request dhcp -clientid tmj0462clr-tmjk01-vip

crsctl set css parameter value
crsctl set css ipmiaddr ip_address
crsctl set css ipmiaddr 192.0.2.244
crsctl set css ipmiadmin ipmi_administrator_name
crsctl set css ipmiadmin scott

crsctl set log {[crs | css | evm "component_name=log_level, [...]"] | [all=log_level]}
crsctl set log crs "CRSRTI=1,CRSCOMM=2"
crsctl set log evm all=2
crsctl set log res "myResource1=3"
crsctl set {log | trace} module_name "component:debugging_level [,component:debugging_level][,...]"
crsctl set log crs "CRSRTI:1,CRSCOMM:2"
crsctl set log crs "CRSRTI:1,CRSCOMM:2,OCRSRV:4"
crsctl set log evm "EVMCOMM:1"
crsctl set log res "resname:1"
crsctl set log res "resource_name=debugging_level"
crsctl set log res "ora.node1.vip:1"
crsctl set log crs "CRSRTI:1,CRSCOMM:2" -nodelist node1,node2
crsctl set trace "component_name=tracing_level,..."
crsctl set trace "css=3"

crsctl setperm resource resource_name {-u acl_string | -x acl_string | -o user_name | -g group_name}
crsctl setperm resource myResource -u user:scott:rwx
crsctl setperm type resource_type_name {-u acl_string | -x acl_string | -o user_name | -g group_name}
crsctl setperm type resType -u user:scott:rwx
crsctl setperm serverpool server_pool_name {-u acl_string | -x acl_string | -o user_name | -g group_name}
crsctl setperm serverpool sp3 -u user:scott.tiger:rwx

crsctl start cluster [-all | -n server_name [...]]
crsctl start cluster -n node1 node2
crsctl start crs
crsctl start ip -A {IP_name | IP_address}/netmask/interface_name
crsctl start ip -A 192.168.29.220/255.255.252.0/eth0
crsctl start resource {resource_name [...] | -w filter | -all} [-n server_name] [-k cid] [-d did] [-env "env1=val1,env2=val2,..."] [-i] [-f]
crsctl start resource myResource -n server1
crsctl start testdns [-address address [-port port]] [-once] [-v]
crsctl start testdns -address 192.168.29.218 -port 63 -v

crsctl status resource {resource_name [...] | -w "filter"} [-p | -v [-e]] | [-f | -l | -g] [[-k cid | -n server_name] [-d did]] | [-s -k cid [-d did]] [-t]
crsctl status resource ora.stai14.vip
crsctl stat res -w "TYPE = ora.scan_listner.type"
crsctl status type resource_type_name [...]] [-g] [-p] [-f]
crsctl status type ora.network.type
crsctl status ip -A {IP_name | IP_address}
crsctl status server [-p | -v | -f]
crsctl status server { server_name [...] | -w "filter"} [-g | -p | -v | -f]
crsctl status server node2 -f
crsctl status serverpool [-p | -v | -f]
crsctl status serverpool [server_pool_name [...]] [-w] [-g | -p | -v | -f]
crsctl status serverpool sp1 -f
crsctl status serverpool
crsctl status serverpool -p
crsctl status serverpool -w "MAX_SIZE > 1"
crsctl status testdns [-server DNS_server_address] [-port port] [-v]

crsctl stop cluster [-all | -n server_name [...]] [-f]
crsctl stop cluster -n node1
crsctl stop crs [-f]
crsctl stop crs
crsctl stop resource {resource_name [...] | -w "filter" | -all} [-n server_name] [-k cid] [-d did] [-env "env1=val1,env2=val2,..."] [-i] [-f]
crsctl stop resource -n node1 -k 2
crsctl stop ip -A {IP_name | IP_address}/interface_name
crsctl stop ip -A MyIP.domain.com/eth0
crsctl stop testdns [-address address [-port port]] [-domain GNS_domain] [-v]

crsctl unpin css -n node_name [node_name [...exit]]
crsctl unpin css -n node1 node4

crsctl unset css parameter
crsctl unset css reboottime
crsctl unset css ipmiconfig

HAS (High Availability Service)
crsctl check has
crsctl config has
crsctl disable has
crsctl enable has
crsctl query has releaseversion
crsctl query has softwareversion
crsctl start has
crsctl stop has [-f]

How do I identify the voting disk/file location?
#crsctl query css votedisk

How to take backup of voting file/disk?
crsctl backup css votedisk

Related Oracle Articles:    cluvfy commands RAC     srvctl commands RAC    crs_stat commands RAC