Release Notes for Couchbase Server 7.1

MB-58335

When memcached restarted, for example, due to an OOM kill, the cluster configuration was not uploaded to memcached. As a result, SDKs did not bootstrap.

When the terse_cluster_info_uploader starts again, the cluster configuration is now refreshed in memcached.

MB-58546

Checkpoint Manager could get stuck when stopping when it had not been started. This caused memory leak.

Checkpoint Manager can now be stopped correctly even when it hasn’t been started.

MB-59336

When stopped during connectivity issues, Checkpoint Manager could hold onto resources longer than usual. This caused a temporary memory bloat.

Checkpoint Manager does not now hold on to resources when it stops during connectivity issues.

MB-59340

A rare timing issue during a pipeline restart prevented the XDCR DCP client from closing. This caused memory leak.

To prevent memory leak, the XDCR DCP client does not now start when the pipeline is stopped.

MB-57400

The computation count for the items remaining DCP/Checkpoint stats exposed to Prometheus was the O(N) function. Where N is the number of items in a checkpoint. This caused various performance issues including Prometheus stats timeouts when checkpoints accumulated a high number of items.

The computation count has now been optimized as O(1).

MB-50868

Janitor run during failover might corrupt the configuration by leaving failed nodes in the bucket server configuration list.

Failed nodes are now filtered out of the bucket server configuration list.

MB-57484

An ns_server process crash meant the Delete Bucket memcached command was not always called before bucket files were deleted later in rebalance. This caused the memcached process to crash repeatedly causing data service downtime.

The Delete Bucket command is now called on memcached before a file is deleted during rebalance. This ensures mencached doesn’t attempt to read the files.

MB-57614

When the REST API attempted to access unsupported stats, such as meta_latency_wt, "stats_reader" crashed on the ns_server side.

ns_server now returns a "not_found" status when the REST API is called with unsupported stats.

MB-56712

XDCR did not process documents with a JSON array and Extended Attributes (XATTRs). When a document contained XATTRs, XDCR checked for XATTRs in transactions, transaction filters were enabled, and XATTRs were not checked.

When documents contain arrays, XATTRs are now checked in the transaction XATTRs, and the document is not prevented from being parsed in an array.

MB-56798

Data streamed from the Data Service over XDCR should always be streamed in order by mutation id. However, in some scenarios, for efficiency, the Data Service streamed records that were not ordered by mutation id. In certain situations, this out-of-sequence-order [OSO] caused performance issues.

OSO mode is now available as a global override to be switched off for any currently deployed replications to avoid performance issues.

MB-56800

Checkpoint Manager created checkpoint records out-of-sequence when many target nodes ran slowly.

Checkpoint Manager now creates checkpoints in sequence when target nodes are slow.

MB-56847

There was an issue in the ns_server stats collector when the Resource Manager checked the pipeline status to obtain the replication priority to provide more resources for higher priority pipelines.

Resource Manager does not now check replication pipeline statuses when all replication pipelines have either the exact same high or low priority.

MB-56849

XDCR on a non-KV node froze in the UI when the replication setting was changed multiple times. This caused the replication reload channel to be full.

The replication reload channel does not now get full when changes are made multiple times to Non-KV nodes.

MB-56850

When target data nodes were undersized or consistently overwhelmed, XDCR memory usages could increase as it retried.

Reallocated memory is now reused instead of generate garbage for GC to clean up.

MB-57131

A Checkpoint Manager Initialization error caused two memory leak types. These were a backfill pipeline and a main pipeline memory leak.

The Pipeline Manager and backfill pipeline have both been modified to prevent the memory leaks.

MB-57184

XDCR Checkpoint Manager instances were not cleaned up under certain circumstances due to timing and networking issues when contacting target, or when an invalid backfill task was fed in as input.

Checkpoint Manager instances are now cleaned up. A flag has been added to check for invalid backfill tasks.

MB-57235

When a replication spec change was made to a non-Data Service node, delete replication hung and caused the node to return an incorrect replication configuration.

XDCR now checks that the node is running the Data Service and handles it correctly.

MB-57413

Running ipv6 only mode + non-encrypted remote resulted in invalid IP addresses being returned leading to connection issues.

A valid IP address is now returned.

MB-57463

StatsMgr stopping could hang due to watching for notifications resulting in stranded go-routines.

Go-routines are now stopped correctly.

MB-57571

When ipv4 only mode was used, and full encryption only had an alternate address configured where the internal address was unresolvable, XDCR resulted in an error when it contacted the target data nodes.

The specific scenario has been fixed so that replication can now proceed.

MB-57788

A legacy race condition where metadata store could cause a conflict was exposed as part of the binary filter improvements.

Legacy race conditions have all been resolved.

MB-56533

Due to how nested dependencies were handled, a sudden rise in memory utilization of the query service on a node caused a memory alert issue. The node did not recover correctly following a restart.

Nested dependencies are now handled appropriately in the ADVISE statement.

MB-56563

A query with multiple filters on an index key, one of which was a parameter, could produce incorrect results. This was caused by incorrectly composing the exact index spans to support the query.

The way in which exact spans are set has been modified to correct this issue.

MB-56579

Covering FLATTEN_KEYS() on an array index generated incorrect results. This was because a modified version of the ANY clause was applied after the index which meant false positives were retained and Distinct scan rows were eliminated.

The ANY filter is now applied on an index scan itself when covering an index scan with flatten keys.

MB-56683

Inter-service read timeout errors were not detected or handled accordingly. User requests consequently failed with timeout errors without retrying with a new connection.

The error handling and retry mechanism has been modified to handle these types of timeout issues and errors.

MB-56727

Under certain circumstances, a query with UNNEST used a covering index scan and incorrect results were returned. Reference to the UNNEST expression should have prevented the covering index from being used for the query as the index did not contain the entire array.

The logic to determine covering UNNEST scans has been changed to not use a covering index scan for such queries.

MB-56937

When an index scan had multiple spans, index selectivity was incorrectly calculated.

Index selectivity for multiple index spans is now correctly calculated.

MB-56940

Incorrect results were returned when a filter contained conditional query parameters. This issue was due to a problem in an OR clause that depended on a named parameter and not a document.

Constant filters in the subterms of the OR clause are now detected and marked. The extra check prevents index aggregation pushdown. For classifying expressions, when there is an OR clause under an AND, processing removes any constant subterms.

MB-56941

A query plan was changed between Server releases. This meant the filter did not update the index when an OR clause pushed variable spans.

The OR clause has been modified to correct this issue.

MB-57024

Incorrect results were returned for a non-IndexScan on a constant false condition. This was due to incorrect handling of a FALSE WHERE clause.

The FALSE WHERE clause is now correctly handled.

MB-57029

Querying system:functions_cache in a multi query node cluster returned incomplete results with warnings. The query result included entries in the local query node, but none from remote query nodes. This was due to a typographical error.

The typographical error has been corrected.

MB-57080

A panic in go_json.stateInString under parsed value functions caused by incorrect concurrent access resulted in the state being freed whilst still in use.

The concurrent access issue has been resolved.

MB-57216

The values returned by the OBJECT_ functions were erroneously pooled and reused by subsequent invocations. Depending on when values were reused, the original results were overwritten.

Pooling has been removed eliminating the chance that values are overwritten.

MB-57316

cbq required a client authentication key file whenever a certificate authority file was used.

cbq now accepts a certificate authority file without a client key file enabling use with username and password credentials.

MB-57680

When appropriate optimizer statistics were used in Cost-Based Optimizer (CBO), for a query with ORDER BY, if there were multiple indexes available for the query, CBO unconditionally favored an index that provided ordering. Such indexes were not always the best ones to use.

CBO now allows cost-based comparison of indexes.

MB-57039

Backup and Restore did not complete successfully when bucket names contained a period or full stop character [.] This was due to a filtering issue where this character was not correctly validated.

Backup and Restore has been fixed to correctly handle any period characters in bucket names.

MB-56195

When indexer was under high memory pressure, queuing 256k mutations added to more memory pressure. For each bucket, indexer queued a minimum of 256k mutations before throttling for memory.

The number of queued mutations has been reduced so that indexer can handle high memory pressure situations much better.

MB-56340

During scaling, an GSI indexer rebalance froze and did not successfully complete. This was because an index snapshot was not correctly deleted and recreated.

A flag now handles snapshots to ensure they are correctly deleted or recreated when indexes are updated during rebalancing.

MB-56341

When the indexer was slow to process mutations, a rare race condition resulted in incorrect book-keeping for the indexer. This meant index builds did not complete.

The race condition no longer happens.

MB-56371

When a partitioned key was based on a secondary field of a document and a delete mutation occurred, the Indexer could not determine which partition the document belonged to. This resulted in delete operations on all partitions.

For partition indexes with document ID as the only partition key, delete mutations are routed only to the partition where the document belongs. This improves the performance of delete and expiration mutations.

MB-56559

The indexer.settings.rebalance.redistribute_indexes flag did not affect partitioned indexes. Partitioned Indexes were by default considered for movement during Rebalance.

The indexer.settings.rebalance.redistribute_indexes flag has been modified to consider partitioned and non-partitioned indexes when restricting the number of index movements during a rebalance.

MB-56604

When alter index updated the replica count, new replicas were not built immediately when the original definition was {defer_build: true}. Existing replicas were built and new replicas were built in the next processing iteration.

New replicas are now built when the replica count is updated for deferred indexes. The status of existing index instances is checked, and if ready, a new build of the instance is triggered.

MB-57135

Watcher threads were created by metadata_provider during rebalance but were not cleaned up.

Threads are now closed after rebalance is finished.

MB-57295

Indexer contained incorrect logic to identify active indexer nodes during a multi-service rebalance. This caused potential downtime and failures in index creation, builds, or other DDL operations.

The information used by TranslatePort has been updated to use the node Services endpoint to correct this issue.

MB-53898

When expired documents were identified during compaction, the Data Service queued a read of the documents' metadata as part of expiry processing. No upper bound was imposed on the size of this queue. This could result in exceeding the Bucket quota for workloads when large amounts of documents expired in a short time.

Metadata reads for TTL processing are not now queued. Instead, they are processed inline. Consequently, Bucket quota is no longer exceeded.

MB-55268

A shared allocation cache (tcache) between buckets resulted in a stats drift. This caused higher-than-normal memory fragmentation.

Dedicated tcaches are now used for buckets. jemalloc has been changed to support increased numbers of tcaches.

MB-55943

Workloads involving bulk data ingestion or Time-To-Live (TTLs) expiring at the same time caused a sudden increase in memory fragmentation.

The defragmenter now runs more frequently to better cope with sudden increases in fragmentation.

MB-56084

A rollback loop affected legacy clients when collections were used and a tombstone newer than the last mutation in the default collection was purged.

The lastReadSeqno is now Incremented when the client is not collection-aware.

MB-56644

In rare cases, after a failover or memcached restart, a replica rollback while under memory pressure might have caused a crash in the Data Service.

Memory pressure recovery logic (Item expelling) is now skipped when replica rollback is in progress.

MB-56970

XDCR or restore from backup entered an endless loop if attempting to overwrite a document which was deleted or expired some time ago with a deleteWithMeta operation. This was due to a specific unanticipated state in memory which increased CPU usage, and connection became unusable for further operations.

deleteWithMeta is now resilient to temporary non-existent values with xattr datatype.

MB-57002

When using .NET SDK on Windows 10 client and client certs were enabled on CB Server, the Data-Service did not establish a connection and client bootstrap failed with a OpenSSL “session id context uninitialized" error.

Data-Service has been updated to disable TLS session resume.

MB-57064

GET_META requests for deleted items fetched metadata in memory which was not evicted in value-eviction buckets.

Metadata items are now cleaned when the expiry pager runs.

MB-57106

DCP clients streamed in OSO backfill snapshots under Magma observed duplicate documents received in the disk snapshot. This happened where the stream was paused and resumed when the resume point was wrongly set to a key already processed in the stream.

OSO backfill in Magma now sets the correct resume point after a pause.

MB-57609

A spurious auto-failover could happen when Magma compaction visited a TTL’d document that was already deleted.

Document not found does not now increment the number of read failures.

MB-57138

A server regression in version 7.1.2 might have caused a cURL request encoding issue.

The default behavior has now been reverted to that in version 7.1.0. In addition, there’s now an optional argument, "url_encode_version", with potential values of [6.6.2, 7.1.0, and 7.2.0]. This argument facilitates the selection of an encoding scheme during upgrades if necessary.

MB-57164

The eventing producer process terminated the eventing consumer process when it did not receive a heartbeat from the consumer process.

The message receiver loop routine now only exits upon receiving a proper termination command.

MB-56957

External collections could not be created using Azure Managed Identity.

Azure dependencies have been updated to correct this issue.

MB-57588

Query results could be unnecessarily converted twice to JSON when documents were large.

The Query result is now converted to JSON once for all documents.

MB-57615

When the Prometheus stats returned from Analytics exceeded four kilobytes, the status code was inadvertently set to 500 (Internal Error), and this resulted in a large number of warnings in the Analytics warning log. Couchbase Server discarded these statistics.

This has been fixed to properly return a 200 (OK) status code when the size of Prometheus stats exceeds 4KiB, allowing these stats to be recorded properly. The warning is not displayed.

MB-57156

Inconsistencies were observed where a single Magma bucket in a database took a long time to warm up.

The seq index scan has been optimized for tombstones of zero value size. Optimization is for look up by key, sequence iteration, and key iteration. Docs of 0 value size are placed in both key index and seq index.

MB-33522

Fix cbupgrade for single node IPv6 clusters

MB-47806

Windows installer always rollbacks during install

MB-44777

The old bucket 'sasl_password' should be effectively removed

MB-44800

The versions REST API should be authenticated

MB-49512

Cleaning up of the cluster fails with "Rebalance exited with reason {buckets_shutdown_wait_failed"

MB-46827

Limit the Checkpoint memory usage

MB-49977

Cannot make persistent change to num nonio/auxio threads

MB-50708

Align roles to updated permissions in memcached

MB-50383

ViewEngine doesn’t handle the case of empty default-collection

MB-51045

Views 8092 REST API leaking version info

MB-51446

On corrupt remote link details in metakv, analytics cluster becomes permanently unusable on restart

MB-19101

Query log format

MB-44757

Support FTS’s docid_regexp mode for N1QL

MB-46802

Mutation fail may not report the error

MB-47366

Public interface documentation on parsing 12009 DML error

MB-48402

Like functions escape character should be optional

MB-33546

Smart Batching Index Builds During Rebalance

MB-46725

Rebalance button not enabled post Quorum Loss failover even when indexing has partitioned indexes

MB-46895

Internal Server error is raised while performing backup on a index node using cbbackupmgr

MB-51196

Index build stuck during rebalance due to large number of pending items

MB-26024

Rebalance optimisations via index file transfer across nodes

MB-41195

Bind only to IPv4 addresses when invoked with IPv4-Only cluster-wide setting

MB-46260

FTS - Apply RBAC only for target collections in a multi-collection index

MB-46978

n1fty to upgrade to blevesearch/sear for verification phase

MB-47017

Support encrypted certificate / key / password - Search

MB-47029

System Event Log - Search

MB-47177

Multiple Root CA Certs - FTS

MB-49188

Search UI should be able to accept queries as objects

MB-49218

Add Croatian language (hr) to the list of supported languages

MB-51059

SEARCH_META().score behaves different from SEARCH_SCORE() in some N1QL queries

MB-45973

Timer not firing after upgrade, worker count change and service crash

MB-46304

Add ability to enable/disable the cURL functionality

MB-48702

Eventing consumes large amount of CPU with no functions.

MB-50936

Summary: Implement defs.CheckMixedModeCallback for mixed mode checks

Any attempt to execute a function with N1QL udfs replicated from a 7.1 node will fail with "no library found in worker" on a 7.0.x node.

Workaround: If possible, all nodes in cluster should be running under version 7.1 or higher.