A newer version of this documentation is available.

View Latest

Function: Advanced Document Controlled Expiry

      +

      Goal: Purge a document automatically based on self-contained start and duration fields.

      • This function advancedDocControlledSelfExpiry demonstrates self-expiry of a document; for example, a user trial.

      • Requires Eventing Storage (or metadata collection) and a "source" collection.

      • Needs a Binding of type "bucket alias" (as documented in the Scriptlet).

      • When documents are created, they will have no expiration value. This function processes the initial mutation to calculate and set the proper TTL.

      • In Couchbase, when using a simple integer expiry value (as opposed to a proper date or time object), the expiration can be specified in two ways:

        • As an offset from the current time. If the absolute value of the expiry is less than 30 days (60 * 60 * 24 * 30 seconds), it is considered an offset.

        • As an absolute Unix time stamp. If the value is greater than 30 days (60 * 60 * 24 * 30 seconds), it is considered an absolute time stamp.

        • As described in Expiration, if a "Bucket Max Time-To-Live" is set (specified in seconds), it is an enforced hard upper limit. As such, any subsequent document mutation (by SQL++, Eventing, or any Couchbase SDK) will result in the document having its expiration adjusted and set to the bucket’s maximum TTL if the operation has:

          • No TTL.

          • A TTL of zero.

          • A TTL greater than the bucket TTL.

      • As we are using Advanced Bucket Accessors setting document expirations (or TTLs) we use a JavaScript Data object.

      • Will operate on any document with type == "trial_customers".

      • Will ignore any doc with a non-zero TTL

      • This is different than setting a TTL on a bucket or a collection which will typically update (or extend) the TTL of a document on each mutation.

      • advancedDocControlledSelfExpiry

      • Input Data/Mutation

      • Output Data/Mutation

      Two variants of this function are available - a 6.6 version that relies on SQL++ and a 6.6.1+/7.0.0+ version (this Function) that directly sets the expiration. You can completely avoid N1QL(…​) and use couchbase.replace(bucket_binding, meta, doc) as the advancedDocControlledSelfExpiry variant is much faster.

      advancedDocControlledSelfExpiry (direct TTL)

      // To run configure the settings for this Function, advancedDocControlledSelfExpiry, as follows:
      //
      // Version 7.1+
      //   "Function Scope"
      //     *.* (or try bulk.data if non-privileged)
      // Version 7.0+
      //   "Listen to Location"
      //     bulk.data.source
      //   "Eventing Storage"
      //     rr100.eventing.metadata
      //   Binding(s)
      //    1. "binding type", "alias name...", "bucket.scope.collection", "Access"
      //       "bucket alias", "src_col",       "bulk.data.source",        "read and write"
      //
      // Version 6.6.1
      //   "Source Bucket"
      //     source
      //   "MetaData Bucket"
      //     metadata
      //   Binding(s)
      //    1. "binding type", "alias name...", "bucket", "Access"
      //       "bucket alias", "src_col",       "source", "read and write"
      
      function OnUpdate(doc, meta) {
          // Filter items that don't have been updated
          if (meta.expiration !== 0) {
              log(meta.id, "IGNORE expiration "+meta.expiration+" !== 0 or "+
                  new Date(meta.expiration).toString());
              return;
          }
      
          // Optional filter to a specic field like 'type'
          if (doc.type !== 'trial_customers') return;
      
          // Our expiry is based on a JavaScript date parsable field, it must exist
          if (!doc.trialStartDate || !doc.trialDurationDays) return;
      
          // Convert the doc's field timeStamp and convert to unix epoch time (in ms.).
          var docTimeStampMs = Date.parse(doc.trialStartDate);
      
          var keepDocForMs = doc.trialDurationDays * 1000 * 60 * 60 * 24 ;
          var nowMs = Date.now();  // get current unix time (in ms.).
      
          // Archive if we have kept it for too long no need to set the expiration
          if( nowMs >= (docTimeStampMs + keepDocForMs) ) {
      
              // Delete the document form the source collection via the map alias
              delete src_col[meta.id];
      
              log(meta.id, "DELETE from src_col to dst_bkt alias as our expiration " +
                  new Date(docTimeStampMs + keepDocForMs).toString()) + " is already past";
          } else {
              var key = meta.id;
              //set the meta.expiration=ttlMs
              var	ttlMs = docTimeStampMs + keepDocForMs;
      
              if (ttlMs !== 0) {
                  log(meta.id, "UPDATE expiration "+meta.expiration+" === 0 set to "+
                      ttlMs+" or " + new Date(ttlMs).toString());
                  // Advanced Bucket Accessors use JavaScript Date objects
                  var expiryDate = new Date(ttlMs);
                  // This is 4X to 5X faster than using N1QL(...) and no need to worry about recursion.
                  var res = couchbase.replace(src_col,{"id":meta.id,"expiry_date":expiryDate},doc);
                  if (!res.success) {
                      log(meta.id,'Setting TTL to',expiryDate,'failed',res);
                  }
              }
          }
      }

      We want to create a test set of four (4) documents, use the Query Editor to insert the the data items (you do not need an index).

      Note, if the today is past 08-25-2021 (MM-DD-YYYY) just change the trialStartDate for the last two records to at least 90 days from now.

        UPSERT INTO `bulk`.`data`.`source` (KEY,VALUE)
        VALUES ( "trial_customers::0", {
          "type": "trial_customers",
          "id": 0,
          "trialStartDate": "08-25-2019",
          "trialDurationDays": 30,
          "note": "this is old will get immeadiately deleted"
        } ),
        VALUES ( "trial_customers::1",
        {
          "type": "trial_customers",
          "id": 1,
          "trialStartDate": "01-27-2020",
          "trialDurationDays": 30,
          "note": "this is old will get immeadiately deleted"
        } ),
        VALUES ( "trial_customers::2",
        {
          "type": "trial_customers",
          "id": 2,
          "trialStartDate": "08-25-2021",
          "trialDurationDays": 30,
          "note": "this will get an exiration set"
        } ),
        VALUES ( "trial_customers::3",
        {
          "type": "trial_customers",
          "id": 3,
          "trialStartDate": "08-26-2021",
          "trialDurationDays": 60,
          "note": "this will get an exiration set"
        } );
      NEW/OUTPUT: KEY trial_customers::2
      
      {
        "id": 2,
        "note": "this will get an exiration set",
        "trialDurationDays": 30,
        "trialStartDate": "08-25-2021",
        "type": "trial_customers"
      }
      
      NEW/OUTPUT: KEY trial_customers::3
      
      {
        "id": 3,
        "note": "this will get an exiration set",
        "trialDurationDays": 60,
        "trialStartDate": "08-26-2021",
        "type": "trial_customers"
      }
      
      We end up with two (2) of the four documents (obviously you may need to adjust the {sqlpp} INSERT in a few months as all the document would be immediately deleted).
      
      * "trial_customers::0" was deleted
      * "trial_customers::1" was deleted
      * "trial_customers::2" has an meta.expiration set for 1632466800 (or 2021-09-24 07:00:00 UTC) in it's metadata
      * "trial_customers::3" has an meta.expiration set for 1635145200 (or 2021-10-25 07:00:00 UTC) in it's metadata