After you establish a failover association, you can monitor its status periodically to ensure that it is functioning properly. You can also delete a failover association when it is not assigned to any DHCP range.
See the following sections on how to manage failover associations:
Under special circumstances, you can manually adjust the configuration of a failover association. For example, when you know in advance that a peer will be out of service for an extended period of time, you can manually set the functional peer in a PARTNER-DOWN mode. This allows the functional partner to assume all leases and be able to allocate addresses to client requests in full capacity. In addition, when you suspect the databases in a failover association are not synchronized, you can consider doing a force recovery (after you consult with Infoblox Technical Support or your Infoblox representative) so the secondary server can completely rebuild its lease table with updates from the primary server.
See the following sections on how to set a peer to the partner-down mode and perform a force recovery:
Modifying Failover Associations
To modify a failover association:
- From the Data Management tab, select the DHCP tab -> Members tab -> Failover Associations -> failover_association check box, and then click the Edit icon.
- The DHCP Failover Association editor contains the following tabs from which you can modify data:
- General: In the Basic tab, modify the fields as described in Adding Failover Associations.
In the Advanced tab, complete the following to modify the port number you use for the failover association:
- Failover Port: Click Override to enter a port number for the failover association. You can use any available port from 1 to 63999. The default is 647 for a new installation and 519 for an upgrade.
- Triggers: Before editing the triggers and timers, ensure that you understand the ramification of the changes. Improper configuration of the triggers can cause the failover association to fail. For information about the fields in the Basic tab, see Adding Failover Associations. The following are the triggers in the Advanced tab:
- Max Response Delay Before Failover(s): Specifies how much time (in seconds) can transpire before a failover occurs when a failover peer does not receive any communication from its peer. This number should be small enough that the transient network failure does not leave the servers out of communication for a long time, but big enough that the servers are not constantly connecting and disconnecting. The default is 60 seconds.
- Max Number of Unacked Updates: Specifies the number of "unacked" packets the server can send before a failover occurs. The default is 10 messages.
- Max Client Lead Time (s): Specifies the length of time that a failover peer can renew a lease without contacting its peer. The larger the number, the longer it takes for the peer to recover IP addresses after moving to the PARTNER-DOWN state. The smaller the number, the more load your servers experience when they are not communicating. The default is 3600 seconds.
- Max Load Balancing Delay (s): Specifies the cutoff after load balancing is disabled. The cutoff is based on the number of seconds since a client sent its first DHCPDISCOVER message. For instance, if one of the failover peers gets into a state where it is busy responding to failover messages but is not responding to other client requests, the other peer responds to the client requests when the clients retry. This does not cause a failover. The default is three seconds.
- Failover Settings: This is valid for Microsoft Management only. Modify failover association settings. For information, see Configuring Failover Associations.
If you modify failover settings from secondary Microsoft server settings, the appliance does not update failover settings on NIOS for the following reasons:
- When DHCP synchronization is disabled for primary Microsoft server, you must enable DHCP synchronization for primary Microsoft server to reflect the settings on NIOS.
- The primary synchronization interval must be completed. For example, consider that you are modifying failover settings from secondary Microsoft server settings where the synchronization interval for primary server is five minutes, and the time interval for the secondary server is one minute. In this case, failover settings are updated on NIOS only after the primary server synchronization interval, which is five minutes.
- Extensible Attributes: Add and delete extensible attributes that are associated with a failover association. You can also modify the values of extensible attributes. For information, see Using Extensible Attributes.
Monitoring Failover Associations
After you configure a failover association, the peers establish a TCP connection for communication. In a normal operational state, they send keepalive messages and database updates every time they grant a lease. However, there are times when the failover association experiences problems and goes into a state other than NORMAL. You can monitor the overall state of a failover association and the individual status of the peers to verify that the servers are operating and communicating properly.
Both peers in a failover association maintain the same DHCP fingerprinting state (enabled or disabled) even when one of the peers fails or becomes operational again. Note that both peers must be in the same Grid for the fingerprinting state to stay the same. For information about DHCP fingerprinting, see About DHCP Fingerprints.
In this panel, you can also modify some of the data in the table. Double click a row of data, and either edit the data in the field or select an item from a drop-down list. Note that some fields are read-only. For more information about this feature, see Modifying Data in Tables.
To monitor the failover association status:
- From the Data Management tab, select the DHCP tab -> Members tab -> IPv4 Failover Associations section. Grid Manager displays the list of failover associations and their overall status.
- To view detailed information about a failover association, select the failover_association check box, and then click the Show Status icon.
- In the Failover Association Status dialog box, Grid Manager displays the overall status of the failover association and the status of both the primary and secondary servers.
The failover association can be in one of the following states:
- OK (green): The failover association is functioning properly.
- DEGRADED (yellow): The failover association is degraded when one of the peers is giving out limited addresses.
- FAILURE (red): The failover association is not functioning, may be because it is not completely configured. The peers are not assigning IP addresses.
For each peer, Grid Manager displays the hostname or IP address, the status, and event date. The peer can be in one of the following states:
- STARTUP: The server is starting up.
- NORMAL: The server is in a normal operational state in which it responds to its load balancing subset of DHCP clients.
- PAUSED: This state allows a peer to inform the other peer that it is going out of service for a short period of time so the other peer can immediately transition to the COMMUNICATIONS-INTERRUPTED state and start providing DHCP service to DHCP clients.
- COMMUNICATIONS-INTERRUPTED: The servers are not communicating with each other. Both servers provide DHCP service to DHCP clients from which they receive DHCP requests.
- PARTNER-DOWN: The server assumes control of the DHCP service because its peer is out of service.
- RECOVER: The server is starting up and trying to get a complete update from its peer and discovers that its peer is in the PARTNER-DOWN state.
- RECOVER-WAIT: The server has got a complete update from its peer and is waiting for MCLT period to pass before transitioning to the RECOVER-DONE state.
- RECOVER-DONE: The server completed an update from its peer.
- POTENTIAL-CONFLICT: The peers are not synchronized due to an administrative error or an incorrect state transition. Check the failover configuration and correct the error.
- CONFLICT-DONE: This is a temporary state that the primary server enters after it received updates from the secondary server when it was in the POTENTIAL-CONFLICT state.
- RESOLUTION-INTERRUPTED: The server responds to DHCP clients in a limited way when it is in this state.
- UNKNOWN: The DHCP server is in an unknown state. The failover association is not functioning properly, may be because it is configured improperly. For example, failover association is not assigned to any DHCP range.
- SHUTDOWN: This state allows a peer to inform the other peer that it is going out of service for a long period of time so the other peer can immediately transition to the PARTNER-DOWN state and completely assume control of the DHCP service.
NIOS does not support PARTNER-DOWN and Force Recovery for a Microsoft DHCP failover association.
Deleting Failover Associations
You cannot delete a failover association if it is currently assigned to a DHCP range. If you want to delete a failover association, ensure that it is not assigned to any DHCP range.
To delete a failover association:
- From the Data Management tab, select the DHCP tab -> Members tab -> Failover Associations -> failover_association check box, and then click the Delete icon.
- In the Delete Confirmation dialog box, click Yes.
The appliance puts the failover association in the recycle bin, if enabled.
Setting a Peer in the Partner-Down State
If one of the peers in a failover association is out of service for an extended period of time, you should consider putting the functional peer in the PARTNER-DOWN state. When you place the functional peer in the PARTNER-DOWN state, it assumes full DHCP services for the networks. Since the functional server may not receive all the updates from its peer, it extends all the leases on the MCLT. Once the following conditions are met, the functional peer provides DHCP services autonomously:
- It has reclaimed all the leases that belonged to its peer.
- The MCLT has passed.
When the peer that is offline comes back online, it synchronizes with the functional peer and reestablishes the communication before it provides DHCP services to the clients.
Before you put a peer in the partner-down state, ensure that the other peer is indeed out of service. If both the primary and secondary servers are operational when you place one of them in the partner-down mode, both servers may stop issuing leases for a minimum of time defined in the MCLT.
To set a peer in the PARTNER-DOWN state:
- From the Data Management tab, select the DHCP tab ->Members tab- > Failover Associations -> failover_association check box.
- Expand the Toolbar and click Set Partner Down.
- In the Set Failover Association Partner Down dialog box, select one of the following:
- Primary: Select this if the secondary server is out of service.
- Secondary: Select this if the primary server is out of service.
4. Click OK.
You cannot place the functional peer in the PARTNER-DOWN state for a Microsoft DHCP failover in NIOS.
Performing a Force Recovery
When the primary and secondary peers are not synchronized, you can perform a force recovery to set the primary server in the PARTNER-DOWN state while putting the secondary server in the RECOVER state. During a force recovery, all leases in the databases are resynchronized. When you perform a force recovery, the secondary server does not serve any DHCP leases for a minimum of the MCLT while it resynchronizes with the primary server. Before you perform a force recovery, consult with Infoblox Technical Support or your Infoblox representative to ensure that the force recovery is appropriate for the situation.
To perform a force recovery:
- From the Data Management tab, select the DHCP tab-> Members tab-> Failover Associations -> failover_association check box.
- Expand the Toolbar and click Force Recovery State.
- In the Force Secondary Peer Recovery State dialog box, click OK.
The appliance synchronizes the databases on the primary and secondary server s.
You cannot place the functional peer in the PARTNER-DOWN state for a force recovery in NIOS.
Recovering DHCP Failover Associations
During a conflict resolution, when the primary peer of the DHCP failover association is in the CONFLICT-DONE state and the secondary peer in the POTENTIAL-CONFLICT state, the secondary peer might experience problems (such as restarting, network outage, etc.) and goes into an invalid state. This results in a deadlock state for the failover association, causing a DHCP service outage. When the failover association is in a deadlock state, you can perform a recovery for the failover association. You can run the recovery for one failover association at a time and when the primary member is in the CONFLICT-DONE state. This feature is supported for Infoblox appliances only and not for any other external DHCP servers.
When the failover recovery is in progress, the DHCP service on both peers are disabled and you cannot enable the DHCP service until the failover recovery is successfully completed. You can view the logs of the failover recovery process in the syslog and infoblox.log file.
To recover a DHCP failover association:
- From the Data Management tab, select the DHCP tab-> Members tab -> Failover Associations -> failover_association check box.
- Expand the Toolbar and click Recovery from Deadlock State.
- In the Failover Recovery Progress dialog box, click Start to start the recovery of the failover association from the deadlock state.
- In the confirmation dialog box, click Yes.
After you start the failover recovery, you cannot revert the changes.
Grid Manager starts the failover recovery and you can view the following information in the Failover Recovery Progress
- Failover association: The name of the failover association.
- Primary: The hostname or IP address of the primary server.
- Secondary: The hostname or IP address of the secondary server.
- Number of leases to be processed: The total number of leases to be processed.
- Number of leases processed: The number of leases that have been processed.
- Current Status: Displays the current status of the failover recovery process. The current status can be one of the following:
- Pending: The failover recovery is initiated for a failover association and the recovery process will start soon.
- Calculating: The appliance calculates the total amount of leases to be processed.
- Applying: The appliance looks for conflicts and tries to resolve the conflicts.
- Completed: The failover recovery is completed successfully.
- Failed: The failover recovery fails.
Grid Manager also displays the reason for the failure if that happens.
After successful completion of the failover recovery, you must restart both the primary and secondary peers to bring them back to the CONFLICT-DONE state.
You can stop the failover recovery operation by clicking Stop in the Failover Recovery Progress dialog box before the recovery process is complete.
If for any reasons the recovery is blocked when the operation is in progress, you can cancel the current operation and start the recovery for the failover association again.