Release Notes

KAFKAC-362: Source The connector can now set Couchbase document metadata headers on Kafka records.

KAFKAC-363: Upgrade Couchbase Java SDK from 3.7.4 to 3.7.5.

KAFKAC-353: Source When streaming from SAVED_OFFSET_OR_NOW, you can configure the connector to immediately save the "now" source offsets for all Couchbase partitions immediately on startup, instead of waiting for events to arrive for a partition before storing that partition’s source offset.

KAFKAC-358: Sink The new couchbase.topic.to.document.id connector config property lets you override the couchbase.document.id property for specific topics. The value is a comma-delimited list of map entries, where each map entry looks like: topicName=${/json/pointer/to/id/field}

KAFKAC-359: The connector now includes provider configuration files for KIP-898: Modernize Connect plugin discovery.

KAFKAC-361: Upgrade Couchbase Java SDK from 3.7.1 to 3.7.4.

KAFKAC-360: Upgrade DCP client from 0.51.0 to 0.52.0.

KAFKAC-352: Source The default event filter, com.couchbase.connect.kafka.filter.AllPassFilter, now excludes events from Couchbase system scopes. (System scopes store internal metadata that is typically uninteresting to developers.)

KAFKAC-347: The connector now writes its metrics to the log (in addition to exposing them via JMX like before). Configure the logging interval by setting the new couchbase.metrics.interval connector property. The default interval is 10 minutes.

KAFKAC-348: Source The connector now captures additional metrics related to Filter and SourceHandler performance.

KAFKAC-350: Upgrade Couchbase Java SDK from 3.6.1 to 3.7.1

KAFKAC-343: Upgrade DCP client from 0.50.0 to 0.51.0. Notably:

KAFKAC-340: Upgrade Couchbase Java SDK from 3.5.2 to 3.6.1.

KAFKAC-337: Sink The connector no longer logs TooManyInstancesDetectedEvent warnings on startup. These warnings were harmless, but annoying.

KAFKAC-334: Upgrade Couchbase DCP client from 0.48.0 to 0.49.0.

KAFKAC-333: Fixed a race condition that could cause the connector to fail at startup with a ConcurrentModificationException.

KAFKAC-327: Sink The couchbase.default.collection and couchbase.topic.to.collection config properties now support qualifying the collection with a bucket name, like my-bucket.my-scope.my-collection. This lets a single instance of the sink connector write to different buckets.

KAFKAC-332: The following connector config options, which were added in previous versions as "uncommitted" API, are now part of the "committed" (stable) API:

KAFKAC-331: Upgrade Couchbase Java SDK from 3.4.7 to 3.5.2.

KAFKAC-330: Upgrade Couchbase DCP client from 0.47.0 to 0.48.0.

JDCP-239: Source Improved reliability of initial startup during a Couchbase Server cluster rebalance, or before a newly-created bucket is ready.

JDCP-240: Source When couchbase.stream.from is set to NOW or SAVED_OFFSET_OR_NOW and there is an inactive partition when the connector starts up, the connector now waits for the partition to become active, instead of starting from the beginning for that partition.

JDCP-241: Source Improved handling of an edge case that could cause the connector to stop streaming for a partition. If a DCP socket connection is lost after receiving a snapshot marker for a partition and before receiving the first item in the snapshot, the connector now recovers instead of failing to resume streaming for that partition.

KAFKAC-307: Source Instead of resolving a DNS SRV address just once when the connector starts up, the connector now passes the DNS SRV address along to the DCP client, so the DCP client can re-resolve it if necessary.

KAFKAC-323: Source If the initial DCP connection fails, the connector no longer throws RejectedExecutionException and no longer leaks memory.

KAFKAC-325: Upgraded Couchbase DCP client from 0.46.0 to 0.47.0.

KAFKAC-324: Upgraded Couchbase Java SDK from 3.4.6 to 3.4.7.

KAFKAC-319: When resuming from an old checkpoint (where the connector’s sequence number is lower than Couchbase Server’s purge sequence number), the connector no longer rolls back to zero. This behavior requires Couchbase Server 7.2 or later.

KAFKAC-322: Upgraded Couchbase DCP client from 0.44.0 to 0.46.0.

KAFKAC-321: Upgraded Couchbase Java SDK from 3.4.3 to 3.4.6.

KAFKAC-311: Source If the source bucket is configured to preserve history, the connector publishes every version of a document it sees within the history window, not just the latest version. See Delivery Guarantees for details. Requires Couchbase Server 7.2 or later.

KAFKAC-316: Upgraded Couchbase DCP client from 0.43.0 to 0.44.0.

KAFKAC-317: Upgraded Couchbase Java SDK from 3.4.1 to 3.4.3.

KAFKAC-303: Source The connector now includes DCP snapshot boundaries in its source offsets. This improves the reliability of resuming from saved offsets.

KAFKAC-302: Source The connector now logs more diagnostic information about lifecycle milestones.

KAFKAC-301: Upgraded Couchbase DCP client from 0.42.0 to 0.43.0.

KAFKAC-305: Upgraded Couchbase Java SDK from 3.3.4 to 3.4.1.

JDCP-232: Fixed a race condition that sometimes caused the connector to fail on startup with the message: java.lang.IllegalStateException: Tried to add duplicate channel.

The Couchbase dcp-client library no longer includes a repackaged version of Jackson. If you have written your own custom Filter, SourceHandler, or SinkHandler component that depends on the DCP client’s repackaged Jackson, search for:

import com.couchbase.client.dcp.deps.

import com.couchbase.client.core.deps.

JDCP-224: Use SASL mechanism PLAIN when authenticating with Couchbase on a secure connection. PLAIN is the fastest mechanism, and the only one that works with LDAP users.

JDCP-217: Support Couchbase clusters that advertise only TLS ports.

KAFKAC-299: Upgrade Couchbase DCP client from 0.41.0 to 0.42.0.

KAFKAC-295 Source The new couchbase.collection.to.topic config property lets you specify an arbitrary mapping from Couchbase scope & collection to Kafka topic. This is useful when the couchbase.topic property is not sufficient because the desired topic name differs from the collection name. Thanks to Shahrzad Haji Amin Shirazi.

KAFKAC-293 Sink N1qlSinkHandler now honors the configured destination scope & collection. Thanks to Shahrzad Haji Amin Shirazi.

KAFKAC-294 Sink The new experimental AnalyticsSinkHandler sends documents directly to the Analytics service. Thanks to Shahrzad Haji Amin Shirazi.

KAFKAC-296 Source In extremely low-traffic environments that ignore a majority of Couchbase events, the new couchbase.black.hole.topic config property may reduce the occurrence of rollback to zero. If a topic is specified, the connector sends a tiny record to this topic for each ignored event, to inform Kafka Connect about the event’s source offset.

KAFKAC-297: Upgraded Couchbase Java SDK from 3.3.0 to 3.3.4.

KAFKAC-298: Upgraded DCP client from 0.40.0 to 0.41.0

KAFKAC-290 When connecting to Capella, it is no longer necessary to configure the Certificate Authority certificate. All you need to do is enable TLS.

KAFKAC-288: Upgraded Couchbase Java SDK from 3.2.5 to 3.3.0.

KAFKAC-283: The connector no longer leaks JMX MBeans on shutdown. The leak was a regression in version 4.0.2.

KAFKAC-284 Source Added a new couchbase.enable.dcp.trace config option for enabling a DCP protocol trace to assist with diagnosing connector issues. Also added couchbase.dcp.trace.document.id.regex to narrow the scope of the trace.

KAFKAC-286: Upgraded Couchbase Java SDK from 3.2.4 to 3.2.5.

KAFKAC-287: Upgraded Couchbase DCP Client from 0.39.0 to 0.40.0.

KAFKAC-279: Upgraded Couchbase DCP Client from 0.37.0 to 0.39.0:

KAFKAC-281: Upgraded Couchbase Java SDK from 3.2.3 to 3.2.4.

KAFKAC-282: Upgraded other dependencies to latest versions.

KAFKAC-275: Sink Added the couchbase.retry.timeout config property. If non-zero, the connector retries write failures until the timeout expires. This is better than simply extending the KV timeout; see the documentation for details.

KAFKAC-276: Upgraded Couchbase Java SDK from 3.2.0 to 3.2.3.

KAFKAC-271: Source The default flow control buffer size is now 16 MB instead of 128 MB. This makes it less likely the source connector will run out of memory under heavy load with the default heap size. The documentation now describes how the couchbase.flow.control.buffer config property affects the Kafka Connect worker’s memory requirements.

KAFKAC-272: Source Upgraded DCP client from 0.36.0 to 0.37.0. This upgrade adds a workaround for MB-48655 so all versions of Couchbase now correctly log DCP diagnostic messages from the source connector.

KAFKAC-269: Upgraded the jsoup library from 1.13.1 to 1.14.2

KAFKAC-266: Source Upgraded DCP client from 0.35.0 to 0.36.0 for compatibility with Couchbase Server 7.0.2.

Apache Kafka 2.4.0 is now the minimum required version. For Confluent Platform users, this corresponds to Confluent version 5.4.0.

The config options for working with Couchbase 7 Scopes and Collections are now part of the "committed" API.

All other "uncommitted" config options are promoted to "committed" status as well.

KAFKAC-257: Sink The connector’s behavior is now completely customizable by implementing the SinkHandler interface. The new couchbase.sink.handler config option specifies the class to use. See the custom extensions example code to see how to implement your own handler.

KAFKAC-263: Source Upgraded DCP client from 0.34.0 to 0.35.0.

KAFKAC-258: Sink Upgraded Couchbase Java SDK from 3.1.3 to 3.2.0.

KAFKAC-261: Sink A connector configured to use SUBDOCUMENT mode with an operation of ARRAY_PREPEND or ARRAY_PREPEND could mistakenly ignore updates if Kafka records targeting the same document arrive in rapid succession. This is now fixed.

KAFKAC-262: Sink A connector configured to use N1QL mode could mistakenly ignore updates if Kafka records targeting the same document(s) arrive in rapid succession. This is now fixed.

KAFKAC-250: Sink You can now configure Couchbase Java SDK Settings in the connector config file. This includes KV timeout durations, Response Time Observability settings, and lots more.

KAFKAC-251: Source Upgraded DCP client from 0.33.0 to 0.34.0.

KAFKAC-211: Source A rollback in Couchbase no longer causes the connector to fail.

KAFKAC-234: The Couchbase root CA certificate can now be read directly from a PEM file; it’s no longer necessary to add it to a Java keystore first. The new couchbase.trust.certificate.path config property points to the PEM file.

KAFKAC-242: Source When the new couchbase.log.document.lifecycle config property is set to true, the connector writes detailed log entries as each document flows through the connector.

KAFKAC-245: Sink Upgraded Couchbase client from 3.1.2 to 3.1.3.

KAFKAC-246: Source Upgraded DCP client from 0.32.0 to 0.33.0. OBSERVE_SEQNO events are now logged at TRACE level instead of DEBUG.

KAFKAC-241: When secure connections are enabled, it is now possible to authenticate with Couchbase using an X.509 certificate instead of a username & password. See couchbase.client.certificate.path and couchbase.client.certificate.password for details. (This feature is added as "uncommitted", meaning it may change without notice.)

KAFKAC-238: Sink Upgraded Couchbase client from 3.0.9 to 3.1.2.

KAFKAC-239: Source Upgraded DCP client from 0.31.0 to 0.32.0. Previously, TLS hostname verification was done only for the Couchbase Java client connection; now the DCP client connection is verified as well. If for some reason you need to disable TLS hostname verification, this is now possible by setting the couchbase.enable.hostname.verification config property to false.

JDCP-183: If an invalid stream offset is detected, the connector will now fail fast instead of potentially corrupting the saved checkpoint.

JDCP-184: Resolved an issue that could cause a flow control deadlock when streaming from a subset of a bucket’s collections or scopes.

KAFKAC-152: Documentation - Added a "Monitoring" page which refers users to the Kafka Connect framework documentation for monitoring connectors via JMX.

KAFKAC-232: Source - Metrics specific to the Couchbase source connector are now exposed via JMX under the kafka.connect.couchbase domain.

KAFKAC-110: Source - Added a gauge that reports the Couchbase connection status.

KAFKAC-231: Source - Upgrade DCP client from 0.30.0 to 0.31.0

KAFKAC-227: Source - Added config property couchbase.xattrs (boolean, defaults to false). If set to true, a custom Filter or SourceHandler may inspect a document’s extended attributes by calling DocumentEvent.xattrs().

KAFKAC-226: Renamed the ZIP archive from couchbaseinc-kafka-connect-couchbase to couchbase-kafka-connect-couchbase (removed the "inc" from "couchbaseinc"). This made it easier to publish the connector on Confluent Hub.

KAFKAC-228: Upgraded the Couchbase Java SDK from 3.0.6 to 3.0.9. Bootstrap performance is improved when specifying custom ports, and the connector no longer logs spurious warnings about being unable to fetch collections manifests.

KAFKAC-229: Upgraded the DCP client from 0.29.0 to 0.30.0, adding support for XATTRs.

KAFKAC-225: Fixed a regression in version 4.0.0 that broke alternate address resolution. The connector now handles DNS SRV and alternate addresses correctly, and can connect to Couchbase Capella or other network environments that use alternate addresses.

KAFKAC-192: Renamed the connector config properties to follow the standard Kafka naming convention ("lowercase.with.dots.between.words"). See the Migration Guide for a comprehensive list of changes, and a shell script that can help upgrade your 3.x config files to use the new property names.

KAFKAC-157: The connector is now packaged as a Confluent Hub component. Confluent Platform users can easily install the connector using the confluent-hub install command. The Quickstart Guide has been updated to show how Apache Kafka users can install the connector.

KAFKAC-167: You can now specify custom ports in the list of Couchbase seed nodes. If you specify a port, it must be the port of the Key/Value service (which defaults to 11210 for unencrypted connections).

KAFKAC-207: Sink - You can now specify an "enhanced durability" requirement with the new couchbase.durability config property. Enhanced durability requires Couchbase Server 6.5 or later.

KAFKAC-197: Sink - Added config property couchbase.topic.to.collection which maps Kafka topics to Couchbase collections. Added config property couchbase.default.collection which is used when a message’s topic is not present in the map.

KAFKAC-209: Source - The connector now evenly distributes the workload among all tasks, instead of trying to minimize the total number of Couchbase connections.

KAFKAC-177: Source - The example config files now use RawJsonSourceHandler and publish Kafka messages whose contents are the same as the Couchbase documents.

KAFKAC-212: Source - If you don’t care about the content of the Couchbase document, you can set the new couchbase.no.value config property to true, and the connector will omit the document content from Kafka messages.

KAFKAC-194: Source - A custom SourceHandler can now set headers on the Kafka record.

KAFKAC-223: Source - The connector is now more responsive to "pause" requests.

KAFKAC-220: Source - Custom Filter and SourceHandler components can now access connector configuration properties. These interfaces now have an init(Map<String, String>) method. The connector config is passed to this method when the component is created.

KAFKAC-222: The custom-extensions example project now includes a CustomFilter class that demonstrates how to read properties from the connector config. This example filter accepts or rejects documents based on a field of the document. The target field and the list of acceptable values are both configurable.

KAFKAC-196: Source - Added couchbase.scope and couchbase.collection config properties that let you stream from a specific scope or set of collections.

KAFKAC-195: Source - Modified the couchbase.topic config property to be a format string that supports ${bucket}, ${scope}, and ${collection} placeholders. This makes it easy to publish to different Kafka topics depending on the Couchbase document’s parent collection. The default value is ${bucket}.${scope}.${collection}.

KAFKAC-171: The couchbase.password config property (previously called connection.password) no longer defaults to an empty string.

KAFKAC-175: APIs deprecated in version 3.x have been removed.

Upgraded Kafka Connect API from 1.0.2 to 2.5.0.

Upgraded Couchbase client from 2.7.13 to 3.0.6.

Upgraded DCP client from 0.26.0 to 0.29.0.

KAFKAC-169: Sink - If two Kafka messages with the same key arrive in rapid succession, it’s no longer theoretically possible for them to be written to Couchbase in the wrong order.

KAFKAC-220: Source - Custom Filter and SourceHandler components can now access connector configuration properties. These interfaces now have an init(Map<String, String>) method. The connector config is passed to this method when the component is created.

KAFKAC-222: The custom-extensions example project now includes a CustomFilter class that demonstrates how to read properties from the connector config. This example filter accepts or rejects documents based on a field of the document. The target field and the list of acceptable values are both configurable.

KAFKAC-197: Sink - Added config property couchbase.topic.to.collection which maps Kafka topics to Couchbase collections. Added config property couchbase.default.collection which is used when a message’s topic is not present in the map.

KAFKAC-207: Sink - You can now specify an "enhanced durability" requirement with the new couchbase.durability config property. Enhanced durability requires Couchbase Server 6.5 or later.

KAFKAC-206: Source - Config property couchbase.connector.name.in.offsets now defaults to false again. This property doesn’t do anything useful, and should only be set to true if you previously had compat.connector_name_in_offsets set to true.

KAFKAC-177: Source - The example config files now use RawJsonSourceHandler and publish Kafka messages whose contents are the same as the Couchbase documents.

KAFKAC-209: Source - The connector now evenly distributes the workload among all tasks, instead of trying to minimize the total number of Couchbase connections.

KAFKAC-212: Source - If you don’t care about the content of the Couchbase document, you can set the new couchbase.no.value config property to true, and the connector will omit the document content from Kafka messages.

KAFKAC-205: Removed the unused couchbase.force.ipv4 config property.

KAFKAC-169: Sink - If two Kafka messages with the same key arrive in rapid succession, it’s no longer theoretically possible for them to be written to Couchbase in the wrong order.

KAFKAC-214: Sink - The Couchbase Java SDK has been updated from 3.0.5 to 3.0.6. As a result, setting couchbase.document.expiration to longer than 30 days now works correctly instead of causing immediate expiration. (This was a regression in 4.0.0-dp.1.)

KAFKAC-203: Source - The 3.x → 4.0 migration script now properly converts the old couchbase.flow_control_buffer property to the new name: couchbase.flow.control.buffer.size.

KAFKAC-204: Source - Fixed the invalid value for couchbase.bootstrap.timeout in the quickstart-couchbase-source.json example config file.

KAFKAC-182: Upgraded Kafka Connect API from 1.0.2 to 2.5.0.

KAFKAC-188: Upgraded Couchbase client from 2.7.13 to 3.0.5.

KAFKAC-189: Upgraded DCP client from 0.26.0 to 0.28.0.

KAFKAC-192: Renamed the connector config properties to follow the standard Kafka naming convention ("lowercase.with.dots.between.words"). See the Migration Guide for a comprehensive list of changes, and a shell script that can help upgrade your 3.x config files to use the new property names.

KAFKAC-196: Source: Added couchbase.scope and couchbase.collection config properties that let you stream from a specific scope or set of collections.

KAFKAC-195: Source: Modified the couchbase.topic config property to be a format string that supports ${bucket}, ${scope}, and ${collection} placeholders. This makes it easy to publish to different Kafka topics depending on the Couchbase document’s parent collection. The default value is ${bucket}.${scope}.${collection}.

KAFKAC-194: Source: A custom SourceHandler can now set headers on the Kafka record.

KAFKAC-157: The connector is now packaged as a Confluent Hub component. Confluent Platform users can easily install the connector using the confluent-hub install command. The Quickstart Guide has been updated to show how Apache Kafka users can install the connector.

KAFKAC-167: You can now specify custom ports in the list of Couchbase seed nodes. If you specify a port, it must be the port of the Key/Value service (which defaults to 11210 for unencrypted connections).

KAFKAC-171: The couchbase.password config property (previously called connection.password) no longer defaults to an empty string.

KAFKAC-173: The couchbase.connector.name.in.offsets config property (previously called compat.connector_name_in_offsets) now defaults to true.

KAFKAC-175: APIs deprecated in version 3.x have been removed.