Backup Service

The Backup Service lets you schedule full and incremental data backups for individual buckets or for all buckets in the cluster. It supports backing up both Couchbase and Ephemeral buckets. The Backup Service also supports scheduled merges of previous backups. You can choose what data to back up by service. For example, you can choose to back up the data for just the Data and Index Services.

You can configure and administer the Backup Service using the Couchbase Server Web Console, the command-line tools, or the REST API.

The Backup Service uses the cbbackupmgr command-line tool to perform backups. You can also directly use this tool to perform backups and merges. Either backup method lets you perform incremental backups and merge incremental backups to deduplicate data. They use the same backup archive structure. You can list the contents of backed up data and search for specific documents no matter how you back up the data.

When choosing whether to use the Backup Service or to directly call cbbackupmgr, consider these differences between these methods:

See cbbackupmgr for more information about using the command-line tool.

When there are multiple Backup Service nodes in the cluster, the Cluster Manager elects one of them to be the leader. The leader is responsible for:

If the leader becomes unresponsive or fails over, the Backup Service stops until a rebalance takes place. During the rebalance, the Cluster Manager elects a new leader. The Backup Service then resumes running on the surviving Backup Service nodes.

To automate backups using the Backup Service, you must create a plan that tells the service what you want it to do. A plan contains the following information:

A repository is a location where the Backup Service can store backup data. You associate a repository with a plan. You must set several options to define the repository, including:

Once you define the repository, the Backup Service performs backups and optionally merges of the data in the bucket or buckets on the schedule in the plan.

After the Backup Service has backed up data, you can inspect it in several ways. You can view the history of backups the Backup Service has performed in a repository. You can also search the repositories for individual documents that have been backed up.

When restoring data from a backup, you can define filters to choose a subset of the data to restore. You can restore data to its original keyspace or apply a mapping to restore it to a different keyspace.

You do not have to restore backed up data to a bucket using the same storage engine as the original bucket. For example, you can restore data backed up from a bucket that used the Couchstore storage engine to a bucket using Magma.

If you no longer need a repository to perform backups, you can archive it. You can still read the backed-up data in an archived repository. However, the Backup Service cannot perform further backups to the repository.

If you delete a repository but do not delete the data it contains, you can import the data back into the cluster. After importing the data, you can read the data, but as with archived repositories, the Backup Service cannot write backups to it.

The Backup Service allows you to schedule automated tasks at intervals as small as one minute. However, you should be cautious about using intervals under fifteen minutes. You must make sure the interval is large enough to allow each task enough time to finish before the next task is scheduled to start.

Several conditions can cause a backup task to take longer than expected. Having many backups in the same repository can make the process of populating the backup’s staging directory slower. Spikes in network latency can also cause a backup to take longer than usual.

The Backup Service runs only a single task at a time. If another instance a task is scheduled to start while a previous instance is still running, the Backup Service refuses to start the new instance. Instead, the instance of the task fails to start. If a backup task is scheduled to start while a different task is already running, the Backup Service queues the new task until the existing task finishes.

A backup task can also fail if the underlying cbbackupmgr process it calls to perform the backup fails. When run directly or by a Backup Service task, the cbbackupmgr tool takes a lock on the repository into which it’s backing up data. This lock prevents any other instance of the cbbackupmgr tool to storing data into the repository. If the instance of cbbackupmgr started by a Backup Service task exits due to a lock on its repository, the backup task fails.

For example, suppose you have a repository whose plan defines two tasks named TaskA and TaskB:

When a task fails to start, the next successful backup task backs up the data it would have backed up.

As explained in the previous section, backup tasks can fail to start if tasks that are already running use the same repository. You have several options to configure your cluster to avoid having backup tasks fail due to these conflicts.

The simplest option is to have a single Backup Service node. This configuration is useful if you have multiple backup tasks that target the same repository. If one task is scheduled to start while another task is running, the Backup Service adds the scheduled task to a queue instead of causing it to fail. One drawback of this configuration is that it reduces resiliency. If the single Backup Service node fails over, then there is no other Backup Service available to handle backups.

If you want greater resiliency for your backups, you can add multiple Backup Service nodes to the cluster. This increases the risk of having backup tasks fail due to overlap if backing up into the same repository.

In either of these cases, you still need to schedule the tasks so that the same task does not overlap with itself.

As explained in the Schedule Merges section, the Backup Service lets you set a schedule for automatically merging previous backups. To schedule merges, you define a past time range within which the Backup Service automatically merges backups.

You set this time range by specifying two offsets, each representing a number of days. The merge_offset_start integer indicates the beginning of the time range and the merge_offset_end indicates its end.

These are offsets from different points in time:

The following diagram shows two examples of settings offsets:

In this diagram, days are numbered from right to left, with today as 0, yesterday as 1, the day before yesterday as 2, and so on. The choice of eight days in the diagram is arbitrary. The Backup Service does not limit the size of the integer when setting the time range.

The diagram contains two examples:

By default, the Backup Service chooses the number of threads it uses based on the number of CPU cores in the node. The Backup Service creates one client connection per thread to connect in parallel to nodes when retrieving their data. The more threads the service uses, the faster it retrieves data. The formula for the number of threads it uses is \(\max(1, cpu\_cores \times 0.75)\) (\(\frac{3}{4}\) the number of CPU cores in the system or 1, whichever is larger).

In some cases, you may find that the default number of threads the Backup Service is using is too high. Depending on resources, a large number of threads can cause performance issues or out-of-memory errors. If your cluster is experiencing these issues during backup, you can change the number of threads a Backup Service node uses. You change this setting using the nodesThreadsMap REST API endpoint. See Manage Backup Service Threads for details on setting the number of threads the Backup Service uses.

Reducing the number of threads the Backup Service uses increases the time it takes to backup data. You may want to experiment with different thread settings to find a balance between resource use and backup duration for your cluster.