Configurate a PowerHA | HACMP using commands line

In this passage,you will learn the steps to configurate the PowerHA (HACMP) using command line:

1.Short introduction about PowerHA(HACMP)

          PowerHA is designed to keep resources highly available with minimum downtime by gathering resources in ways that allow multiple IBM System p servers to access them.PowerHA manages disk, network, and application resources logically, passing control to individual machines based on availability and preference. From a systems administration point of view, the main concept behind PowerHA is tokeep everything as redundant as possible to ensure that there is high availability at all levels.

2.Key PowerHA terms

        The following terms are used throughout this article and are helpful to know when discussing PowerHA:

Cluster:A logical grouping of servers running PowerHA.

Node: An individual server within a cluster.

Network:Although normally this term would refer to a larger area of computer-to-computer communication (such as a WAN), in PowerHA network refers to a logical definition of an area for communication between two servers. Within PowerHA, even SAN resources can be defined as a network.

Boot IP: This is a default IP address a node uses when it is first activated and becomes available. Typically—and as used in this article—the boot IP is a non-routable IP address set up on an isolated VLAN accessible to all nodes in the cluster.

Persistent IP: This is an IP address a node uses as its regular means of communication. Typically, this is the IP through which systems administrators access a node.

Service IP: This is an IP address that can "float" between the nodes. Typically, this is the IP address through which users access resources in the cluster.

Application server:This is a logical configuration to tell PowerHA how to manage applications, including starting and stopping applications, application monitoring, and application tunables. This article focuses only on starting and stopping an application.

Shared volume group:This is a PowerHA-managed volume group. Instead of configuring LVM structures like volume groups, logical volumes, and file systems through the operating system, you must use PowerHA for disk resources that will be shared between the servers.

Resource group:This is a logical grouping of service IP addresses, application servers, and shared volume groups that the nodes in the cluster can manage.

Failover: This is a condition in which resource groups are moved from one node to another. Failover can occur when a systems administrator instructs the nodes in the cluster to do so or when circumstances like a catastrophic application or server failure forces the resource groups to move.

Failback/fallback: This is the action of moving back resource groups to the nodes on which they were originally running after a failover has occurred.

Heartbeat:This is a signal transmitted over PowerHA networks to check and confirm resource availability. If the heartbeat is interrupted, the cluster may initiate a failover depending on the configuration.

3.Configuration a two-node PowerHA cluster

Step 1. Install the PowerHA software
        You can purchase this software from IBM directly; the file sets all start with the word cluster. Use theinstallp command to install the software, much like any other licensed program package (LPP).

Step 2. Edit some flat files
       Put all of the IP addresses associated with the cluster—boot（here,boot IP including the boot and  standby IP) persistent, and service—into each /etc/hosts file on each node of the cluster. Dothe same with the/usr/es/sbin/cluster/etc/rhostsfile. 

       Verify that the server hostnames match the appropriate IP addresses;

       the server's hostname should also match with the persistent IP address.

Step 3. Configure the boot IP addresses
       Run the smitty chinet command, and set the boot IP addresses for each network adapter. Make sure that you are able to ping and connect freely from node to node on all respective networks. Also, double-check to make sure that the default route is properly configured. If it isn't, run smitty tcpip, go into the Minimum Configuration menu, enter the default route for the primary adapter, and press Enter.

Step 4. Make application start and stop scripts
       Create two simple Korn shell scripts—one that starts an application and one that stopsan application. Keep these scripts in identical directories on both nodes.

Step 5. Define the cluster
       Run the command:
     smitty cm_config_an_hacmp_cluster_menu_dmn
Then, define the cluster, including naming it appropriately.

Step 6. Define the nodes
     Run the command:
     smitty cm_config_hacmp_nodes_menu_dmn

Define every node within the cluster on both nodes.

Step 7. Define the networks
       Run the command:
     smitty cm_config_hacmp_networks_menu_dmn


This defines one network per Ethernet adapter. I prefer to use the Pre-defined option as opposed to the Discovered path, but that is up to your discretion. Check the subnet masks for consistency.

Step 8. Define the boot IP addresses
     Run the command:
    smitty cm_config_hacmp_communication_interfaces_devices_menu_dmn

     This defines the boot IP addresses on the respective network adapters. This address should be the same IP addresses you used in step 3. Make sure you define these addresses within the proper respective PowerHA-defined network.

Step 9. Define the persistent IP addresses
     Run the command:

smitty cm_config_hacmp_persistent_node_ip_label_addresses_menu_dmn

This defines the persistent IP addresses, again paying attention to pick the proper respective PowerHA-defined network.

Step 10. Define the service IP addresses
     Run the command:
    smitty cm_config_hacmp_service_ip_labels_addresses_menu_dmn


This defines the service IP addresses.

Step 11. Perform a discovery and reboot
       By this point, the nodes should have the ability to communicate with each other and keep the information stored in the nodes' Object Data Managers (ODMs) in sync.     Make the nodes within the cluster communicate with each other by running the command:
      smitty cm_extended_config_menu_dmn


Select the Discover PowerHA-related Information from Configured Nodes option, and check for errors to fix.Generally, rebooting each node can clear up any minor problems, and this is a good point to test restarting each server anyway.

Step 12. Define the resource group
     Run the command:
     smitty cm_hacmp_extended_resource_group_config_menu_dmn


Define the resource group. Then, perform these steps:
Select all participating nodes in the cluster.
Set the Startup Policy to Online On First Available Node.
Set the Fallback Policy to Never Fallback.
This setting prevents the resources from going back to the original server when it is brought up, which is a wise thing to do.

Step 13. Create a shared volume group
       Note: Run this command only on one node.
    Run the smitty cl_vg command, and create a shared volume group. When you create a shared volume group, you only need to select one of the nodes, because the disk is shared.

Step 14. Create a heartbeat disk
        First, run the smitty cl_convg command, and then select Create a Concurrent Volume Group with Data Path Devices. Choose one node and the target disk. Next, run the command:
    smitty cm_config_hacmp_communication_interfaces_devices_menu_dmn


Repeat step 7, except this time, select the Discovered option and the target disk.

Step 15. Define an application server
     Run the command:
    smitty cm_cfg_app_extended


This defines an application server for an application that PowerHA will manage. Use the scripts you created in step 4.

Step 16. Configure the resource group
    Note: Run this command on only one node.
     Run the command:
    smitty cm_hacmp_extended_resource_group_config_menu_dmn


Select the Change/Show Resources and Attributes for a Resource Group option. Then, perform these steps:
Select the appropriate service IP addresses.
Select the appropriate shared volume groups and heartbeat disk.
Select the appropriate application servers.

Step 17. Perform a synchronization

     Synchronize the cluster configuration. Run the command:
    smitty cm_ver_and_sync


Set Automatically correct errors found during verification? to Interactive. Correct any problems along the way.

Step 18. Start the cluster
      At this point, the cluster is ready to start.On one of the nodes, run thesmitty clstart command, and pick that particular node. My preference is not to have the cluster start on reboot, because if there is a PowerHA-related problem on startup, it can be difficult to troubleshoot it. After the node comes up with the resources available, start the cluster on the other node. 

Step 19. Perform a test failover
       The best way I have found to test PowerHA’s adaptability is to reboot the active node and let things fail over naturally while running thetail –f /tmp/hacmp.out command on the other node to watch as things go over. Or, run the command:
    smitty cl_resgrp_move.node_site


Select the Move Resource Groups to Another Node option.

Step 20. Perform failure testing
     If you really want to make sure your cluster is solid, perform testing by literally removing cables and seeing how the resources move back and forth. The more you test, the more reliable your cluster will be.