Failover

There are two basic types of failover: graceful and hard.

Graceful failover must be manually initiated. Hard failover can be manually initiated. Hard failover can also be initiated automatically by Couchbase Server: this is known as automatic failover. The Cluster Manager detects the unavailability of a node, and duly initiates a hard failover, without administrator intervention.

Note that when a node is failed over (as opposed to removed), some replica vBuckets are lost from the surviving nodes; since some are promoted to active status, and are not replaced with new replica-copies. By contrast, removal creates new copies of those replica vBuckets that would otherwise be lost. This maintains the cluster’s previous level of data-availability; but results in greater competition for memory resources, across the surviving nodes.

Ideally, after any failover, rebalance should be performed. This is especially important when a Data Service node has been failed over, since the rebalance will ensure an optimal ratio of active to replica vBuckets across all the remaining Data Service nodes.

Hard failover is performed after a node has failed. It can be initiated either by administrative intervention, or through automatic failover.

When automatic failover is used, the Cluster Manager handles both the detection of failure, and the initiation of hard failover, without administrative intervention: however, the Cluster Manager does not identify the cause of failure. Following failover, administrator-intervention is required, to identify and fix problems, and to initiate rebalance, whereby the cluster is returned to a healthy state.

If manual failover is to be used, administrative intervention is required to detect that a failure has occurred. This can be achieved either by assigning an administrator to monitor the cluster; or by creating an externally based monitoring system that uses the Couchbase REST API to monitor the cluster, detect problems, and either provide notifications, or itself trigger failover. Such a system might be designed to take into account system or network components beyond the scope of Couchbase Server.

When failover has occurred, and active vBuckets have thereby been replaced through the promotion of replicas, the resulting imbalance in the ratio of active to replica vBuckets on the surviving nodes should be corrected, by means of rebalance.

The variations in vBucket numbers and ratios that occur due to outage, failover, and rebalance are demonstrated by the following tables.

Table 1 shows the disposition of data across a cluster of four nodes. One bucket, of 31,591 items, has been configured with three replicas. The ratio of active to replica items is therefore 1:3. (Note that some numbers in the table are approximated, and are therefore shown in italics.)

Table 2 shows the result of failing over node 4.

Active and replica vBuckets for the failed over node, 4, are still counted, but are not available (and so are marked here with asterisks). To compensate, replicas on nodes 1 to 3 have been promoted to active status; this being evident from their modified numbers and ratios. For example, on node 1, the number of active items is now raised (from its former total of 7,932) to 11,000; while the number of replica items is now lowered (from its former total of 23,600) to 20,500.

Subsequent rebalance corrects the ratios, as shown by Table 3, below.

Ratios on nodes 1 to 3 are now 1:2, indicating that rebalance has reduced the number of replicas for the bucket from 3 to 2, in correspondence with the reduced node-count.

For further examples of rebalance in the context of node removal, see Removal.

the cluster’s Master Services determine how efficiently the cluster handles a failover. They manage operations with cluster-wide impact such as failover, rebalance, and adding and deleting buckets. The Master Services are also called the Orchestrator.

At any given time, only one instance of the Master Services is in charge. The Master Service instances negotiate among themselves to identify and elect one of them to be the active Master Service. If the elected instance becomes unavailable, another instance takes over.

Having the node that’s hosting the active Master Services fail over causes greater latency than usual. The remaining Master Service instances must renegotiate among themselves to determine which becomes the new active Master Service before handling the failover. Having the Master Services co-located with a highly subscribed services such as the Data Service can result in unnecessary latency when failing over the node and ensuring data availability.

To limit failover latency, the Master Services should be co-located on a node with lowest overhead due to services. When choosing a node to host the active Master Service, Couchbase Server weighs the impact of each service running on each node. The services are broken into several tiers, listed here from least to greatest impact:

Each tier is assigned a cost ten times higher than the previous tier. For example, Couchbase Server considers a node running Analytics, Indexing, and Search services more capable of handing the active Master Service role than a node running just the Data Service.

If one is available, Couchbase Service always chooses an the cluster’s Master Services to host the active Master Service. Arbiter nodes do not run any services, and are therefore perfect candidates for hosting the Master Service.

Node removal uses the rebalance process to remove a node from a cluster in a controlled fashion. It creates on the remaining nodes new copies of replica vBuckets that would otherwise be lost when the selected node is taken offline. See Removal for a conceptual description of node-removal. For practical steps, see Remove a Node and Rebalance.