Cisco NAC Profiler Server Command Line Reference

This chapter includes the following topics:

•Overview

•Service Profiler Command Set

•Profiler Server Database Operations

•Upgrading Cisco NAC Profiler Server Software

•HA Pair Operations

•Returning Profiler Server to Factory Defaults

•Changing the LINUX beacon and root Account Passwords

Overview

Most configuration and management tasks for Cisco NAC Profiler are performed via the web-based user interface. However, some functions are performed via the command line on the Profiler Server. This chapter outlines several command line operations that may need to be performed on the Profiler Server.

Note Prior to performing any of the following Command Line tasks, Cisco strongly recommends that you backup the current database, as described in Database Backup.

Service Profiler Command Set

The Cisco NAC Profiler Server command line provides a "service profiler" command set that enables several system-level functions to be executed from the LINUX command line of the Profiler Server. The service profiler commands are available only to the root user. Table 15-1 describes the available commands in the service profiler command set.

HA System Considerations for Service Profiler Commands

Cisco NAC Profiler Servers implemented as HA pairs have additional considerations when using service profiler commands. The Secondary system in an HA pair will always show the status of all installed modules as 'Not Running.' This is the normal state for the Secondary appliance. The HA protocol is maintaining database synchronization and in the event of failure of the Primary, the modules will be started in conjunction with the failover.

In addition, the 'start' option is not available on Secondary appliances in an HA pair. If service profiler start is entered at the command line of an appliance in HA mode in the Secondary state, the following message will be displayed:

               This system is in HA mode and will not be started

While the system is running in HA mode, the Profiler service on the Secondary member of the pair should be left to the control of the HA protocol and not manipulated via the command line.

Profiler Server Database Operations

The Profiler Server utilizes a PostgreSQL database to store all system configuration and endpoint data. PostgreSQL is a powerful, enterprise-class relational database system that strongly conforms to ANSI-SQL 92/99 standards.

Because the system configuration and endpoint data is stored in a single database, system backup and restore consists of creating a backup copy of the Profiler database and moving it to an off-system repository for safe keeping should the Cisco NAC Profiler system need to be rebuilt. In case of catastrophic failure of the appliance running the Server module in a Cisco NAC Profiler system, the recovery model is to restore the Profiler Server from media, then restore the Profiler Server database to the most recent successful backup to restore the entire system to the state it was in previous to the failure. The Server module will perform an automatic copy of the Profiler database each day at 3:00AM system time of the appliance running the module. These daily backups can be found in the /backup directory in compressed (.gz) format. As outlined later in this chapter, the .gz compressed format is the format that the database restore scripts expect when restoring a backed-up database to a system. The Server module will maintain the backup directory so that database backups older than 30 days are automatically deleted from the backup directory.

Note It is highly recommended that the Cisco NAC Profiler Server database backups are moved off the appliance and stored with other system backups in accordance with file backup best practices.

Database Backup

In addition to the automated backups performed by the system, it is also possible to make a backup copy of the Profiler database at any time. In release 2.1.8 and later, a backup copy can be made and transferred off the system through the Profiler GUI. Navigate to the Utilities tab, and select System Summary. At the bottom of the System Summary, three buttons are displayed: Display Server Log, Download DB Dump, and Collect technical logs.

If these buttons are not displayed in the System Summary, the Profiler version is pre-2.1.8. Refer to the procedure described in Database Backup (Versions Earlier Than 2.1.8) to make a backup copy of the Profiler Server database and transfer it off the Profiler Server appliance.

Refer to Release Notes for Cisco NAC Profiler, Release 2.1.8 for complete details on features, enhancements and system upgrade procedures.

Figure 15-1 System Summary Displaying DB Backup Button

To initiate the backup of the database, select the Download DB button which results in the browser displaying the download file dialog which will allow the designation of an off-appliance location to save the database backup copy to. Specifying a path for the file will result in the database being saved to a compressed file and copied to the specified location. By default, the filename for the backup copy is 'beacondb.gz'.

Database Backup (Versions Earlier Than 2.1.8)

In pre-2.1.8 systems, the backup of the database and copy to an off-appliance location must be performed manually. Follow the steps outlined below:

1. Create an SSH session to the system hosting the Server module (For HA systems, this should be the VIP/Service Address of the HA pair) as the beacon LINUX user.

2. Enter the following command to make a backup copy of the database:

pg_dump | gzip > beacondb.gz

3. This will create the database backup file 'beacondb.gz' in the /home/beacon directory

4. Use SCP to copy the database backup to another system for safe keeping should the configuration and endpoint data need to be restored to the Cisco NAC Profiler system.

Profiler Server Database Restore

A backup copy of the Profiler database can be restored to a Profiler Server at any time.

Because the database contains both the system configuration and the endpoint data, any changes made to the system configuration since the backup was created will be lost after a database restore on a Profiler Server.

The Profiler database restore procedure is different depending on whether the Profiler Server is in standalone or HA mode. Before restoring a database to a Profiler system, determine first if it is a standalone or HA system, then use the appropriate procedure to restore the Profiler database:

•Database Restore on Standalone (non-HA) Systems

•Database Restore on HA Systems

Database Restore on Standalone (non-HA) Systems

With release 2.1.8 and later, restoring the Profiler database to a standalone system can be performed through the execution of a single script. Follow these steps to restore the database to a 2.1.8 system.

1. Copy the backup file to be restored to the /backup directory on the system running the Server module via SCP

2. Initiate a SSH session to the system running the Server module, and change directory to /usr/beacon/scripts (cd /usr/beacon/scripts)

3. Run the following script to restore the selected database backup on the standalone system:

./restoreDB.pl /backup/beacondb.gz

where beacondb.gz is the filename of the backup

4. From the GUI, execute an Apply Changes -> Update Modules to restart the system using the restored database.

For database restore on pre-2.1.8 standalone systems (indicated by the above named script not being present in the /usr/beacon/scripts directory), utilize the following steps to drop and create the database manually, then restore the backup copy to it.

1. Copy the backup file to be restored to the /backup directory on the system running the Server module via SCP

2. Initiate a SSH session to the system running the Server module as user beacon, the execute the following commands in the order specified:

a. dropdb beacon

b. createdb beacon

c. gunzip -c /backup/beacondb.gz | psql

where beacondb.gz is the filename of the backup

3. From the GUI, execute an Apply Changes -> Update Modules to restart the system using the restored Profiler database.

Database Restore on HA Systems

Release 2.1.8 contains several usability and maintenance enhancements to the HA functionality, including automation of the database restore process.

Note Prior to restoring the database for HA Profiler Servers, the Profiler Servers must be upgraded to release 2.1.8 or later. Refer to Upgrading Cisco NAC Profiler Server Software.

The Cisco NAC Profiler HA protocol includes a database synchronization function that ensures that the system configuration and endpoint data maintained in the database is kept in sync on both members of the pair. Therefore, the process to restore the Profiler Database on an HA pair requires the restore on the Primary, and reliance on the synchronization process to update the secondary appliance database.

Note In order to ensure that the database restore is performed on the primary, the restore process outlined in this section should be performed on the VIP/Service IP for the HA pair.

Follow these steps to restore a Profiler database on an HA pair:

Step 1 Copy the backup file to be restored to the /backup directory on the primary system via SCP to the VIP/Service IP address of the HA pair

Step 2 Initiate a SSH session to the VIP/Service IP address, and change directory to /usr/beacon/scripts (cd /usr/beacon/scripts)

Step 3 Run the following script to restore the selected database backup on the primary appliance of the HA pair:

./restoreDB-HA.pl /backup/beacondb.gz

where beacondb.gz is the filename of the backup

Step 4 From the GUI for the VIP/Service IP address, execute an Apply Changes -> Update Modules to restart the HA system using the restored database.

Weekly Database Maintenance

The Profiler Server performs necessary database maintenance each week to ensure the integrity of the system automatically. The database maintenance requires a brief stop and restart of the Server module in order to perform these functions. By default, this weekly maintenance occurs automatically on Sunday at 2:00AM based on the system clock of the system running the Server module. This event is logged in the Server logs, is normal behavior and is not a cause for concern.

Upgrading Cisco NAC Profiler Server Software

Updates to the Cisco NAC Profiler software are available from the Cisco Software Download support website at http://www.cisco.com. Cisco.com registration is required and customers must have a valid maintenance contract. Upgrades for the Cisco NAC Profiler Server are released as a single file, in compressed format (.zip).

Along with the upgrade package, an MD5 checksum is posted to ensure file integrity. Verify the MD5 checksum prior to using the upgrade package to ensure that the upgrade package has not become corrupted.

The software upgrade package will include all necessary files for upgrading the Profiler Server software, underlying components, and the Profiler database. The upgrade is performed by a single script, and the script will automatically detect the operating mode of the appliance and upgrade the system accordingly.

For further details on upgrading Cisco NAC Profiler systems, refer to the Release Notes for Cisco NAC Profiler, Release 2.1.8.

Upgrading Standalone Systems

Upgrading the Profiler Server software on standalone Cisco NAC Profiler systems is a straightforward process. The upgrade script that is integral to the upgrade package upgrades all installed components automatically as needed. Follow the steps below to upgrade standalone Profiler Servers.

1. SCP the upgrade package .zip file downloaded from the Cisco Systems support website to the /home/beacon directory

2. SSH to the system being upgraded, and elevate to root user using the su - command.

3. Change directory to /home/beacon (cd /home/beacon), and verify the MD5 checksum of the upgrade package against the checksum specified for the file on the support website. Use the following command to generate the checksum of the file on the target system:

md5sum ProfilerUpgrade-xx.xx.xx-yy.zip

This command will calculate and display the checksum of the file to the terminal on the appliance so it can be checked against the one supplied with the file.

4. Unzip the upgrade package (unzip ProfilerUpgrade-xx.xx.xx-yy.zip). This will uncompress the files required for upgrade, and create a new subdirectory in named ProfilerUpgrade-xx.xx.xx-yy in the beacon/home directory.

5. Change directory to the ProfilerUpgrade directory created when the upgrade package was unzipped.

6. The directory should include a script named upgrade.pl. Execute the upgrade script by entering the following command:

./upgrade.pl

7. During the upgrade process, several messages may be sent to the terminal indicating progress of the upgrade as installed components are upgraded. When the update script completes successfully, the Cisco NAC Profiler service(s) running on the system will be restarted, at the following message displayed:

To modify the configuration of the Profiler, user 'service profiler config'

Followed by the return of the system prompt.

8. Verify the successful upgrade of the system by entering the service profiler status command. The output will include the current version of the Cisco NAC Profiler system, and should indicate the Running status for the installed module(s) on the system, as in the example below.

[root@ProfilerHA1 ~]# service profiler status

Profiler Status

   Version: Profiler-2.1.8-29

  o Server      Running

  o Forwarder   Not Installed

  o NetMap      Not Installed

  o NetTrap     Not Installed

  o NetWatch    Not Installed

  o NetInquiry  Not Installed

  o NetRelay    Not Installed

Upgrading HA Pairs

The procedure for upgrading the software on a HA pair is performed on the Secondary appliance in the pair first, and then on the Primary. In the process of the upgrade, the system that was the Secondary prior to the upgrade will take over the functions of the Primary, similar to what would occur in the event of the failure of the Primary.

Caution Upgrading the Cisco NAC Profiler software on a Profiler Server HA pair should be completed on both members of the pair sequentially in a single operation. Do not leave the pair with the first appliance upgraded and delay the upgrade of the other member of the pair.

If it is desirable to return the HA pair back to its state previous to the upgrade, failover of the pair will be necessary to force the member that was Primary prior to the upgrade back to that state. See Forcing HA Failover on an Profiler Server.

Follow these procedures to upgrade the Cisco NAC Profiler software on a HA pair.

1. Determine which state (e.g., Primary, Secondary) each appliance in the HA pair to be upgraded is currently in as described in Determining which Appliance is Primary. Note the management interface (eth0) IP address of the Secondary appliance.

2. SCP the upgrade package .zip file downloaded from the Cisco Systems support website to the /home/beacon directory of the Secondary appliance.

3. SSH to the IP address of the Secondary system that is being upgraded, and elevate to root user using the su - command.

4. Change directory to /home/beacon (cd /home/beacon), and verify the MD5 checksum of the upgrade package against the checksum specified for the file on the support website. Use the following command to generate the checksum of the file on the target system:

md5sum ProfilerUpgrade-xx.xx.xx-yy.zip

This command will calculate and display the checksum of the file to the terminal on the appliance so it can be checked against the one supplied with the file.

5. Unzip the upgrade package (unzip ProfilerUpgrade-xx.xx.xx-yy.zip). This will uncompress the files required for upgrade, and create a new subdirectory in named ProfilerUpgrade-xx.xx.xx-yy in the beacon/home directory.

6. Change directory to the ProfilerUpgrade directory created when the upgrade package was unzipped.

7. The directory should include a script named upgrade.pl. Execute the upgrade script by entering the following command:

./upgrade.pl

8. During the upgrade process, several messages may be sent to the terminal indicating progress of the upgrade as installed components are upgraded. When the update script completes successfully, the Cisco NAC Profiler service(s) running on the system will be restarted, at the following message displayed:

To modify the configuration of the Profiler, user 'service profiler config'

Followed by the return of the system prompt.

9. Verify the successful upgrade of the system by entering the service profiler status command. The output will include the current version of the Cisco NAC Profiler system, and should indicate Running status for the installed module(s) on the system.

This completes the upgrade of the software on the Secondary appliance. Proceed by copying the upgrade package to the other appliance in the HA pair, and performing the upgrade process on the appliance that was Primary at the outset of the upgrade procedure.

Once the second appliance has been successfully upgraded, both members of the HA pair are now at the new revision. The original Secondary is now the Primary, and vice versa.

If it is desirable to return the HA pair back to the state it was in prior to the upgrade (original Primary now Secondary returned back to Primary state), follow the procedure in Forcing HA Failover on an Profiler Server.

HA Pair Operations

The Cisco NAC Profiler Server supports HA configuration. The HA protocol can be configured at system startup using the procedures outlined Chapter 4, "Installation and Initial Configuration", or HA can be added to an operational standalone system using the procedure outlined in this section at any time. There are several operations that may need to be performed on an HA system from time to time. This section provides guidance on the management of an HA Profiler Servers from the command line.

Determining which Appliance is Primary

At any given time, only one member of an HA pair is running the Cisco NAC Profiler processes and serving as the Primary appliance in the HA pair. The Primary maintains the master Profiler database for the Cisco NAC Profiler HA pair and maintains database synchronization on the Secondary. The Primary appliance will hold the VIP/Service IP for the HA pair. To determine which appliance is currently the Primary and which is Secondary, perform the following steps to determine which appliance in the pair is currently holding the VIP/Service IP for the pair:

1. SSH to the VIP/Service IP for the HA pair as user beacon

2. Issue the following command: /sbin/ifconfig -a to display the configuration of the interfaces on the appliance currently utilizing the VIP for the HA pair. An excerpt of the output from running this command on an appliance is shown below:

[beacon@BeaconHA1 ~]$ /sbin/ifconfig -a

eth0      Link encap:Ethernet  HWaddr 00:04:23:E2:AD:EE

          inet addr:10.10.0.221  Bcast:10.10.0.255 Mask:255.255.255.0

          inet6 addr: fe80::204:23ff:fee2:adee/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:2188142 errors:0 dropped:0 overruns:0 frame:0

          TX packets:942333 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:303102816 (289.0 MiB)  TX bytes:105503829 (100.6 MiB)

          Base address:0x3020 Memory:b8820000-b8840000

eth0:0    Link encap:Ethernet  HWaddr 00:04:23:E2:AD:EE

          inet addr:10.10.0.220  Bcast:10.10.0.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

3. The interface named 'eth0' shows the IP address assigned to the management interface of the appliance that is currently serving as the Primary for the pair. The presence of the eth0:0 virtual interface is indicative of this appliance currently holding the VIP/Service IP address of the HA pair.

4. Establish an SSH session as user beacon to the IP address of the eth0 interface of the Primary as determined in steps 2 & 3 above.

5. Enter the command cd /usr/beacon/sql

6. Run the Check Master Status script by entering ./chk_status_master.sh

7. On the Primary appliance in the HA pair, the result of this script should be as follows:

is master

This indicates the appliance is the current Primary and is maintaining the master Profiler Server database for the HA pair.

Verifying HA Pair Operation

Verification of the proper functioning of the Cisco NAC Profiler HA protocol should be accomplished prior to failing over the system to ensure the protocol is operating normally.

This section provides instructions for verifying the operation of the HA protocol: heartbeat and protocol (slon) operation and database synchronization. The HA protocol will update the Profiler Server database on the Secondary (Slave) system continuously, once the HA initiation sequence is completed properly. Verification of database synchronization is the best test to ensure that the HA protocol is operating normally for a given Profiler Server HA pair.

Check for Heartbeat and SLON Processes

The first check to verify proper operation of the HA process is to verify that both members of the pair are running the heartbeat and SLON processes which are required for normal HA operation. Complete the following steps to ensure these processes are running on both members of the HA pair:

1. Initiate an SSH session with both the Primary and Secondary appliances to their respective interface (eth0) IP address and elevate to root access.

2. Issue the following command on both appliances to determine the status of the heartbeat service:

service heartbeat status

The command should return a result such as the following on both the current Primary and Secondary members of an HA pair:

heartbeat OK [pid 20960 et al] is running on beaconha1 [beaconha1]...

3. Verify that both the Primary and Secondary have the required two slon processes running by issuing the following command:

ps aux | grep slon

This command should show a similar result on both members of an HA pair if the slon processes are running normally:

[root@BeaconHA1 ~]# ps aux | grep slon

root      4646  0.0  0.0   3880   676 pts/1    S+   12:43   0:00 grep slon

beacon   20686  0.0  0.0  67468   932 ?        Sl   07:47   0:00 /usr/bin/slon -d 
0 -p /usr/beacon/working/slon.pid -s 1000 beacon_cluster dbname=beacon user=beacon 
password=beacon

beacon   21089  0.0  0.0   4840   904 ?        S    Dec26   0:00 /usr/bin/slon -d 
0 -p /usr/beacon/working/slon.pid -s 1000 beacon_cluster dbname=beacon user=beacon 
password=beacon

[root@BeaconHA1 ~]#

If both of the preceding checks result in the desired output, proceed with the next step to verify database synchronization on the Secondary appliance. If either the heartbeat or slon processes are not running as indicated above on either member of the HA pair, proceed with the process outlined in the section entitled Repairing a HA Configuration later in this chapter.

Check Database Synchronization on Secondary

To verify that the database synchronization is occurring properly on an HA pair, follow these steps to ensure the database on the Secondary system was created properly and is being regularly updated.

When performing this procedure on a newly configured HA pair, it is essential to wait several minutes after the completing the HA setup process to ensure that the protocol has had ample time to complete the initial database synchronization on the Secondary. The procedure below checks the status of the database tables that are synchronized last, therefore it is important to wait several minutes before using this check on newly configured HA pairs.

1. Determine the IP address of the eth0 (management) interface of the Secondary system in the HA pair, and initiate an SSH session to this system as the beacon user. If you are not sure which appliance in the pair is the Secondary, follow the procedures outlined above to determine which appliance is currently the Primary system.

2. Enter the psql command to initiate the PostgreSQL interactive terminal (enter the Profiler Server database password if requested).

Note PostgreSQL interactive terminal sessions must be closed manually upon completion of database operations. Always be sure to close the interactive terminal session by entering the command \q when complete with the procedures outlined below.

3. The table called 'beacon_component' is among the last to be synchronized on the Secondary member of an HA pair. Query this table on the Secondary member of the HA pair to ensure that the table is present and being updated regularly by using the following SQL query at the interactive terminal prompt:

select name,status_time from beacon_component;

The following is example output from the execution of this step on an operational Secondary system in a Profiler Server HA pair:

[beacon@BeaconHA2 sql]$ psql

Welcome to psql 8.1.8, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms

       \h for help with SQL commands

       \? for help with psql commands

       \g or terminate with semicolon to execute query

       \q to quit

beacon=# select name, status_time from beacon_component;

     name     |        status_time

--------------+----------------------------

 Server       | 2007-12-26 11:11:26.302466

 BeaconHA2-ni | 2007-12-26 11:26:30.722751

 BeaconHA2-fw | 2007-12-26 11:26:48.721669

 BeaconHA2-nm | 2007-12-26 11:26:50.721551

 BeaconHA1-nt | 2007-12-26 11:26:52.684777

 BeaconHA1-fw | 2007-12-26 11:26:53.684616

 BeaconHA1-nw | 2007-12-26 11:26:54.012347

 BeaconHA1-nr | 2007-12-26 11:26:54.685568

 BeaconHA1-nm | 2007-12-26 11:26:54.688583

 BeaconHA1-ni | 2007-12-26 11:26:59.687283

 BeaconHA2-nt | 2007-12-26 11:27:22.752656

 BeaconHA2-nw | 2007-12-26 11:27:23.371504

 BeaconHA2-nr | 2007-12-26 11:27:23.758521

(13 rows)

As the example shows, the output of executing this command on the Secondary system shows all component modules on both members of the HA pair, and the time of the last status message the system has processed for each module, which should be near the system time on the Primary system.

If the error message "Error: relation "beacon_component" does not exist" results from the query, this is indicative of the HA protocol on the pair being incorrectly configured and that the database schema has not been set up correctly. Follow the procedure for Repairing a HA Pair Configuration outlined later in this chapter to re-initialize the HA configuration.

4. Wait several minutes, and execute the query again. The status time should be increasing between queries. This indicates that the HA process is functioning normally, and that the Secondary system is ready to become Primary should the current Primary appliance fail.

5. When verification is complete, be sure to terminate the PostgreSQL interactive terminal session by entering the /q command at the interactive terminal command prompt. (The system prompt should change from beacon=# to the normal prompt [beacon@Hostname sql]$ when successfully ending the terminal session.

If the processes outlined in this section fail to verify that the HA protocol is operating normally, the HA configuration of the pair can be reset using the procedure in the section entitled Repairing a HA Configuration later in this chapter. Following this procedure will remove and restart the HA configuration from the beginning and should restore normal HA operations to an HA pair.

Adding HA to an Operational Standalone Profiler Server

An operational Profiler Server may have an HA peer added to it at any time. In the most common scenario, an operational Profiler Server is to be converted to HA pair operation through the addition of a second appliance. Typically, the appliance that is already operating will become the initial Primary member of the HA pair, and a new, previously unconfigured appliance will be added as the Secondary appliance, set up for that role via the startup scripts that initiate upon the first power-up of the appliance.

This procedure is specifically for the case of adding an HA peer to an operational standalone system. For a new installation of an HA pair, follow the procedures outlined in Chapter 4, "Installation and Initial Configuration".

If the Profiler Server appliance that will be added as an HA pair has been configured and used previous to its employment as a member of an HA pair, that appliance must first be re-imaged to ensure that all configuration and data is removed from the system. Contact Cisco Systems Technical Support for instructions and necessary software to re-image the appliance prior to proceeding.

Note Ensure that all passwords (e.g., LINUX, Profiler Server database, and Web UI admin) for the Primary system are known. When configuring the Secondary member of the HA pair, all passwords on both systems must match exactly.

1. Prior to beginning the process, ensure that the following parameters are determined and readily available:

•VIP/Service IP address to be used

•Hostname of the Secondary appliance being added as an HA peer

•Local HA Network: the first three octets of a private network IP address (e.g., 192.168.1) to be used for the heartbeat network between the 2 appliances (eth1 interfaces).

•HA Authentication Key: a text-string to be utilized by the appliances to authenticate. The HA Shared Key must be entered identically (case sensitive) on both appliances in order for the relationship to be established.

•HA External Ping Host: a host IP address of another network device, preferably on the same subnet as the HA pair that will respond to ICMP echo requests from the Profiler Server appliances. The Profiler Server appliances will ping this device regularly to ensure that they still have network connectivity as a measure to detect the failure of their network interface.

2. Install the crossover cable connecting the eth1 interfaces of the existing appliance (Primary) and the appliance being added as an HA peer. Prior to beginning the remaining steps, ensure that the NIC LEDs are indicating link between the appliances (e.g., left LED on eth1 interface on both appliances solid amber). Lack of proper link indication is indicative of a problem with the cable used to create the heartbeat connection. Ensure that link is established before continuing.

3. Begin the configuration of the HA pair with the configuration of the Primary appliance first. Initiate an SSH or console session with the Primary appliance, and elevate to the root user.

4. Enter cd /root/.BeaconSetup, then ./SetUpHA to initiate the HA portion of the appliance startup scripts on the Primary being sure to answer 'yes' when asked if this appliance is the Primary. The script will prompt then prompt for the entry of each of the parameters specified in #1 above.

5. Complete the configuration of the new appliance being added as the Secondary member of the HA pair. Ensure that all passwords for the Secondary system are configured to be the same as those set on the Primary.

6. During the HA-specific portion of the startup script, ensure that 'no' is selected when the script asks if this will be the Primary, and ensure that the HA parameters such as VIP, Local HA Network, and HA authentication key are configured identically to what was configured in step #4 above on the Primary.

7. Wait several minutes after the completion of the configuration of the Secondary, then utilize the procedure outlined in the section of this chapter entitled Verifying HA Pair Operation to verify operation of the HA protocol. Once verification is completed successfully, failover may be tested by simulating failure of the current primary as described in the section of this chapter entitled Forcing an HA Pair to Failover.

Repairing a Profiler Server HA Configuration

From time to time it may be necessary to remove and reinstall the HA service on an Profiler Server. There are several scenarios which may cause the HA configuration to fail initially, for example the heartbeat network not being in place between the eth1 interfaces on the members of the pair, or Profiler Server database passwords not being identical, as two common examples. When it becomes apparent via the HA pair operation verification procedure describer earlier, or through some other means that the HA protocol is not operating normally, follow the procedure below to first remove the HA protocol on both members of the pair beginning with the Secondary, and then reinitializing the HA configuration on both members.

1. Initiate an SSH or console session with the Secondary appliance first. Elevate to the root user, and enter the following command to remove the HA protocol from the Secondary member of the pair:

/root/.resetProfiler  removeHA

2. Initiate an SSH or console session with the Primary appliance, and use the same command in #1 above to remove the HA configuration from the Primary appliance.

3. Ensure that the heartbeat network is properly configured with a crossover cable between the eth1 interfaces of both members, with link indicated. Verify that all LINUX, database and web UI passwords are identical on both appliances.

4. Prior to continuing the process, ensure that the following parameters specific to the HA pair are verified and readily available:

•VIP/Service IP address to be used

•Hostname of the Secondary appliance being added as an HA peer

•Local HA Network: the first three octets of a private network IP address (e.g., 192.168.1) to be used for the heartbeat network between the 2 appliances (eth1 interfaces).

•HA Authentication Key: a text-string to be utilized by the appliances to authenticate. The HA Shared Key must be entered identically (case sensitive) on both appliances in order for the relationship to be established.

•HA External Ping Host: a host IP address of another network device, preferably on the same subnet as the HA pair that will respond to ICMP echo requests from the Profiler Server appliances. The Profiler Server appliances will ping this device regularly to ensure that they still have network connectivity as a measure to detect the failure of their network interface.

5. Reconfigure the HA protocol beginning with the Primary first. As the root user, issue the following commands on the Primary:

cd /root/.BeaconSetup 

./SetUpHA

6. Complete the HA setup script for the Primary HA appliance utilizing the parameters determined in step 4 above.

7. Reconfigure the HA protocol on the Secondary appliance. As the root user, issue the following commands on the Secondary:

cd /root/.BeaconSetup 

./SetUpHA

8. Complete the HA setup script for the Secondary HA appliance utilizing the parameters determined in step 4 above.

9. Wait several minutes after the completion of the configuration of the Secondary, then utilize the procedure outlined in the section of this chapter entitled Verifying HA Pair Operation to verify operation of the HA protocol. Once verification is completed successfully, failover may be tested by simulating failure of the current primary as described in the section of this chapter entitled Forcing an HA Pair to Failover.

Forcing HA Failover on an Profiler Server

It may be necessary to force a Profiler Server HA pair to failover that is to initiate the transfer of Primary member duties to the Secondary to ensure that the failover capability is fully operational. This may be desirable when the HA system is being tested for example, or at anytime it is determined that the Primary duties should be shifted to the other appliance in the HA pair.

As is described in the Configuration Guide, the Cisco NAC Profiler HA protocol is designed to protect the Profiler Servers implemented as an HA pair from two potential failure modes:

1. Complete failure of the current Primary as indicated by the loss of heartbeat

2. Loss of network connectivity by the Primary and determination that the Secondary has better network connectivity. The latter failure mode is detected by the inability to ping the "ping host" specified in the HA configuration. Note that this is optional, and may not be configured in all implementations.

The preferred method to force the failover of an HA pair is to determine which appliance is currently the Primary, and disabling the heartbeat protocol on that appliance temporarily. After the failover, heartbeat can be re-established and the HA pair returned to normal operation with Primary duties remaining with the appliance that was in the Secondary role at the outset.

When performing this procedure it is essential that either after the initial configuration of an HA pair, or after an induced failover within the HA pair, that enough time is allowed to elapse to allow database synchronization to complete fully prior to failing the system back. Time required for this process to occur is dependent on database size, but in most cases allowing 15-30 minutes to elapse after initial configuration or a forced failover is recommended prior to failing the system over again. Repeatedly forcing an HA pair to fail over without providing ample time for the system to stabilize will result in undesirable behavior and may result in the necessity to remove and re-configure the HA protocol in order to return the system to stable operation.

Follow the procedure below to force the failover of an HA pair:

1. SSH to the eth0 interface of the current Primary appliance in the HA pair that is to be failed over as user beacon. If the current Primary is not known, utilize the procedure in this chapter for determining which appliance is Primary. The SSH session should be to the eth0 interface IP and not the VIP so that the session will remain active through the failover.

2. Change directory to /usr/beacon/sql, and run ./chk_status_master.sh to verify that the system you are currently on is Primary (script returns "is master").

3. Switch user to root via su -

4. SSH to the eth0 interface of the other appliance in the pair, the current Secondary which will become Primary upon the failover.

5. Change directory to /usr/beacon/sql, and run ./chk_status_master.sh to verify that the system is in fact currently the Secondary (script returns "is slave").

6. When ready to failover the pair return to the SSH session on the current Primary, then enter the following command to temporarily stop the heartbeat from the Primary to the Secondary which will in turn induce the desired failover:

service heartbeat stop

7. Wait several seconds then return to the SSH session to the former Secondary, which now should be the current Primary. Verify that is the case by running ./chk_status_master.sh to verify that the system is now the Primary (script returns "is master").

8. On the former Primary which now should be the Secondary, run the following command as root to start the heartbeat again:

service heartbeat start

9. Exit back to the beacon user then run the ./chk_status_master.sh to verify that the system is currently the Secondary (script returns "is slave"). This indicates that the system successfully failed over.

At this juncture the system should now be operating in HA mode after the swap of the Primary duties. Full verification of the operation of the HA protocol can be verified using the procedure outlined earlier in this chapter. If it desired to fail the system back, the procedure above can be repeated after ample time for database synchronization is allowed as outlined earlier in this section.

Temporarily Disabling HA

High Availability services can be temporarily disabled on an active HA pair. This process can be used to disable failover but not remove the base HA configuration from the members of the pair. To temporarily disable HA, follow these steps:

1. Identify the current Secondary member of the pair, using the procedure specified in this chapter if necessary. Establish an SSH session to the system, and elevate to root access (su -).

2. Issue the following command to disable the HA system on the Secondary:

/root/.resetProfiler disableHA

3. The system will display the following message and request confirmation:

[root@BeaconHA2 ~]# /root/.resetProfiler disableHA

This script will disable the HA system but not remove the base configuration.

If you wish to disable the HA system please type 'yes':

4. Type 'yes' to continue with disabling the HA system. The following messages are displayed as the HA service is disabled on the Secondary:

Cleaning up DB synch files

Stopping the HA

Stopping heartbeat.  This may take up to three minutes.

Stopping High-Availability services:

[  OK  ]

HA disabled on this system.

HA *must* be disabled on the alternate host to prevent

the DB from caching all changes until the drive fills up.

'/root/.resetProfiler restartHA' will restart the HA.

Note the message about disabling HA on the other member of the pair. It is important to also disable HA on the Primary to prevent the Primary from continuing to run the HA service and attempting to keep the database on the Secondary updated after the HA service has been disabled. Without normal communications between the members of the pair, the Primary will cache uncompleted database sync attempts causing several tables in the database to grow large quickly. This can be prevented by disabling HA on the Primary as outlined below.

5. Repeat the procedure to disable HA services on the Primary: SSH to the eth0 IP address of the Primary, elevate to root, and run the /root/.resetProfiler disableHA script on the Primary. Enter 'yes' to continue and observe the shutdown of HA services on the Primary.

6. To restore the HA services using the existing base configuration, begin with the system that was Primary prior to disabling HA for the pair. As root, enter the following command:

/root/.resetProfiler restartHA

This will result in the following message being displayed:

[root@BeaconHA1 ~]# /root/.resetProfiler restartHA

This script will restart the HA system from the base configuration.

If you wish to restart the HA system please type 'yes':

Type 'yes' to continue and observe the following messages as the service restarts 
on the Primary member of the pair:

Restarting the HA

Stopping heartbeat.  This may take more than three minutes.

Stopping High-Availability services:

[  OK  ]

Starting High-Availability services:

2007/12/28_15:27:51 INFO:  Resource is stopped

[  OK  ]

HA *must* be running/restarted on the alternate host to prevent

the DB from caching all changes until the drive fills up.

[root@BeaconHA1 ~]#

7. Repeat the same procedure on the Secondary to restart HA services on the other member of the HA pair, verifying the successful restart of the service on the Secondary.

The HA pair is now restored to normal operation.and HA operation can be verified using the procedures outlined in this chapter

Permanently Removing HA Configuration from Profiler Server

If it becomes necessary to remove HA services permanently from a Profiler Server appliance, use the following procedure to completely remove the base HA configuration from the appliances that are currently operating in HA mode. This might be necessary when it is desirable to return an operating Profiler Server HA pair back to a single appliance, stand alone mode.

Note that at the completion of this procedure, the appliance that was Primary for the pair at the outset is now configured to operate as the standalone for the Cisco NAC Profiler system after removal of HA. The appliance that was the Secondary member of the pair, will require reconfiguration prior to re-employment of the appliance. Preferably the appliance should be re-imaged to ensure that the existing configuration and database are completely cleared. Contact Cisco Systems support for assistance with re-imaging a Profiler Server appliance.

1. SSH to the appliance eth0 IP address and elevate to root access. If this procedure is being performed on an active HA pair, it is best to begin with the Secondary appliance.

2. Run the following script to permanently remove the HA base configuration:

/root/.resetProfiler  removeHA

3. Perform the steps above on the Primary appliance to remove the base HA configuration on the Primary.

4. Restart the system using the service profiler restart command.Upon the restart, the former Primary member of the pair is now acting as a standalone Profiler Server.

5. Remove the crossover network cable connecting the appliances previously in the HA pair configuration.

6. If the appliance that was the Secondary prior to the removal of the HA configuration is to be re-used, it is highly recommended that the appliance have the Cisco NAC Profiler system software re-installed on that appliance to ensure that all configuration and database data is removed before reconfiguration.

Returning Profiler Server to Factory Defaults

In some cases it may be desirable to delete all configuration information on a given system to return it to a "factory default" state. For example, when a serious error in configuration is made and it is desirable to start from a clean configuration this procedure will stop, remove and replace the Profiler, allowing the startup scripts to be run again to correct the configuration issues.

Note Note that this procedure is not intended for use on systems that have been configured and used in lab or production environments. The proper way to return systems that have been configured and used is to re-image the appliance. Contact Cisco Systems technical support for instructions and necessary software to re-image a Profiler Server. In both cases all configuration parameters and database information will be removed.

The following procedure will restore a Profiler Server to factory defaults during the configuration process:

1. Initiate a console session to the system to be restored to factory defaults and elevate to root access. (SSH can be used as well, as long as the IP parameters of the appliance will remain unchanged.)

2. Enter the following command to initiate the factory default script:

/root/.resetProfiler factory

3. When the script initiates, it will display the following message at the terminal and requires user input to continue:

This script will stop, remove, and replace the Profiler. Its intended use is to 
clean up an incorrectly configured installation. It should not be used on a system 
that has already been installed and used in a production network.

If you wish to reset the Profiler please type 'yes':

4. To proceed, type yes at the colon and press enter.

5. At the completion of the script, the command prompt will return. Logout of the system entirely, and then re-login as root. (The LINUX passwords for root and beacon will not be changed by the factory default script). The appliance startup scripts will initiate with the following message at the terminal upon logging back in as root.

Welcome to the NAC Profiler Endpoint Profiler.

Hit any key to create the initial configuration (or ^c to exit):

6. Press enter to continue with the configuration of the appliance.

Changing the LINUX beacon and root Account Passwords

Each Cisco NAC Profiler Server appliance has two LINUX accounts used for command-line operations with the appliance.

The password for the LINUX accounts used on a Profiler Server appliance can be changed using the following procedure:

1. Start a console or SSH session with the target appliance as root user (elevate to root using su - if using SSH) and enter the following command:

•passwd beacon - to change the beacon password

•passwd root - to change the root password

Note If a new password containing a dictionary word is entered, the system will post a warning suggesting that a stronger password be used. Either enter a stronger password or re-enter the original password to override the warning and change the password.

Identifying Stale Network Devices in Cisco NAC Profiler System Configuration

Network Devices added to the Cisco NAC Profiler system configuration when the NetMap module is in use will be regularly polled via SNMP according to the parameters specified in the Server module configuration. It is possible for Network Devices to become unreachable by their assigned NetMap module, for example if the SNMP community string is changed on the device without being changed in the Cisco NAC Profiler configuration. The Table of network devices (from the Configuration tab, select Network Devices -> List Network Devices) displays the 'Last Scan' date and time for each device in the configuration. In large environments however, it is often difficult to identify network devices that have not been scanned recently.

The Cisco NAC Profiler system includes a script that can be run to find devices in the configuration that have failed to poll for greater than three days. To find stale network devices in the Cisco NAC Profiler configuration, perform the following procedure on the Profiler Server.

Note On HA systems, perform the following on the Primary by initiating an SSH session to the VIP of the Profiler Server pair.

1. SSH to the management interface (eth0) of the standalone Profiler Server running the Server module for the system (VIP for HA pairs).

2. Change directory to /usr/beacon/scripts (cd usr/beacon/scripts)

3. Located in that directory is a script named testDevices.pl. The script will accept as an argument, the Profiler Server database password if it has been changed from the default of 'profiler'.

4. Run the script by entering ./testDevices.pl [profilerdb pass]

If the following is displayed at the command line:

DBI connect('dbname=beacon','beacon',...) failed: FATAL:  password authentication 
failed for user "beacon"

 at ./testDevices.pl line 17

Can't connect to db

Then the password supplied (or the default if a password was not specified as an argument) is incorrect. Determine the Profiler Server DB password, and re-run the script.

5. The script identifies any device that has not been polled in more than three days; if there are no network devices that have not been polled in three days, the script will not have any output and the command prompt will return.

6. If there are network devices that have not been polled in more than three days, the script will first attempt to contact the device via SNMP, and then ping the device to determine if the device can be contacted directly by the Profiler Server. For devices that fail both tests, the following message is displayed:

Timeout: No Response from 10.50.0.1

BAD 10.50.0.1 NO PING