Recover a Node and Rebalance

After a node has been failed over, it can be recovered: that is, added back into the cluster from which it was failed over, by means of the rebalance operation.

Understanding Recovery

After failover has occurred, before the failed over node has been rebalanced out of the cluster, the node can be recovered, and thereby reintegrated into the cluster. This is useful in circumstances where, following failover, the unhealthy node has been fixed, and is therefore now assumed fit for re-addition.

There are two options for recovery, which are full and delta. These are explained in Recovery.

Examples on This Page

The examples in the subsections below show how to perform both full and delta recovery on the same failed over node in the same cluster; using the UI, the CLI, and the REST API respectively. The examples assume:

The examples assume:

A two-node cluster already exists, as at the conclusion of Join a Cluster and Rebalance.
The cluster has the Full Administrator username of Administrator, and password of password.

Recover a Node with the UI

Proceed as follows:

Access the Couchbase Web Console Servers screen, on node 10.142.181.101, by left-clicking on the Servers tab in the left-hand navigation bar. The display is as follows:
To see further details of each node, left-click on the row for the node. The row expands vertically, as follows:
To initiate failover, left-click on the Failover button, at the lower right of the row for 101.142.181.102:

The Confirm Failover Dialog now appears:

Two radio buttons are provided, to allow selection of either Graceful or Hard failover. Graceful is selected by default.
Confirm graceful failover by left-clicking on the Failover Node button.

Graceful failover is now initiated. A progress dialog appears new the top of the screen, summarizing overall progress.

For server-level details of the graceful-failover process, see the conceptual overview provided in Graceful Failover.

When the process ends, the display is as follows:

This indicates that the graceful failover has successfully completed. A rebalance is required to complete the reduction of the cluster to one node. Additionally, the Add Back: Full Recovery and Add Back: Delta Recovery buttons are displayed, towards the left-hand side of the row:
Select one of the two available forms of recovery, by left-clicking the corresponding button. Note that full and delta recovery are described in Recovery. If you select full, by left-clicking on the Add Back: Full Recovery button, the row for 10.142.181.102 is displayed as follows:

The row specifies REBALANCE to finish full recovery: therefore, left-click the Rebalance button to apply full recovery.

Similarly, left-clicking on the Add Back: Delta Recovery displays REBALANCE to finish delta recovery. Recovery can be aborted, by left-clicking on the CANCEL ADD BACK button.
Left-click on the Rebalance button. Whichever form of recovery you have chosen, full or delta, is performed.

Note that if rebalance fails, notifications are duly provided. These are described in Rebalance Failure Notification. See also the information provided on Automated Rebalance-Failure Handling, and the procedure for its set-up, described in Rebalance Settings.

Recover a Node with the CLI

For a node to be recovered with the CLI, it must first be failed over. Perform graceful failover as documented in Graceful Failover with the CLI, as follows:

couchbase-cli failover -c 10.142.181.101:8091 \
--username Administrator \
--password password \
--server-failover 10.142.181.102:8091

To recover the failed-over node, use the recovery command, in either of the following ways:

Perform full recovery:

couchbase-cli recovery -c 10.142.181.101:8091 \
--username Administrator \
--password password \
--server-recovery 10.142.181.102:8091 \
--recovery-type full

Perform delta recovery:

couchbase-cli recovery -c 10.142.181.101:8091 \
--username Administrator \
--password password \
--server-recovery 10.142.181.102:8091 \
--recovery-type delta

Whichever option you have chosen, if it completes successfully, the command produces the following output:

SUCCESS: Servers recovered

The server must now be rebalanced back into the cluster, as follows:

couchbase-cli rebalance -c 10.142.181.101:8091 \
--username Administrator \
--password password

During rebalance, progress is displayed as console output:

Rebalancing
Bucket: 01/01 (travel-sample)                    60714 docs remaining
[=====                                                        ] 4.56%

If successful, the command returns the following:

SUCCESS: Rebalance complete

For more information on rebalance, see the command reference at rebalance. For more information on recover, see the command reference at recovery.

Recover a Node with the REST API

For a node to be recovered with the REST API, it must first be failed over. Perform graceful failover as documented in Graceful Failover with the REST API, as follows:

curl -v -X POST -u Administrator:password \
http://10.142.181.101:8091/controller/startGracefulFailover \
-d 'otpNode=ns_1@10.142.181.102'

To recover the failed over node, use the recovery command, in either of the following ways:

Perform full recovery:

curl -u Administrator:password -v -X POST \
http://10.142.181.101:8091/controller/setRecoveryType \
-d 'otpNode=ns_1@10.142.181.102' \
-d 'recoveryType=full'

Perform delta recovery:

curl -u Administrator:password -v -X POST \
http://10.142.181.101:8091/controller/setRecoveryType \
-d 'otpNode=ns_1@10.142.181.102' \
-d 'recoveryType=delta'

The server must now be rebalanced back into the cluster. Use the /controller/rebalance URI, as follows:

curl -u Administrator:password -v -X POST \
10.142.181.101:8091/controller/rebalance \
-d 'knownNodes=ns_1@10.142.181.101,ns_1@10.142.181.102'

For more information on /controller/startGracefulFailover, see Setting Graceful Failover. For more information on /controller/setRecoveryType see Setting Recovery Type. For more information on /controller/rebalance, see Rebalancing Nodes.