What's New

• JDBC Multitable Consumer origin enhancement - You can now optionally define a schema exclusion pattern to exclude some schemas from being read. The schema exclusion pattern uses a Java-based regular expression, or regex.

• Kudu Lookup processor enhancements:
  • You can now configure the Maximum Number of Worker Threads property to limit the number of threads that the processor uses.
  • You can now configure an Admin Operation Timeout property to determine how many milliseconds to allow for admin-type operations, such as opening a table or getting a table schema.

• Kudu destination enhancements:
  • You can now configure the Maximum Number of Worker Threads property to limit the number of threads that the destination uses.
  • You can now configure an Admin Operation Timeout property to determine how many milliseconds to allow for admin-type operations, such as opening a table or getting a table schema.

• Data Collector now includes a SPARK_KAFKA_VERSION environment variable that is set to 0.10 by default in the Data Collector environment configuration file - sdc.env.sh or sdcd.env.sh. Do not change this environment variable value. This variable is used only when you run cluster streaming mode pipelines on a Cloudera CDH cluster.

• Directory origin enhancements - The origin includes the following enhancements:
  • The Max Files in Directory property has been renamed to Max Files Soft Limit. As the name indicates, the property is now a soft limit rather than a hard limit. As such, if the directory contains more files than the configured Max Files Soft Limit, the origin can temporarily exceed the soft limit and the pipeline can continue running.

    Previously, this property was a hard limit. When the directory contained more files, the pipeline failed.

  • The origin includes a new Spooling Period property that determines the number of seconds to continue adding files to the processing queue after the maximum files soft limit has been exceeded.

• Einstein Analytics destination enhancement - The Append Timestamp to Alias property is now disabled by default for new pipelines. When disabled, the destination can append, delete, overwrite, or upsert data to an existing dataset. When enabled, the destination creates a new dataset for each upload of data.

  The property was added in version 3.1.0.0 and was enabled by default. Pipelines upgraded from versions earlier than 3.1.0.0 have the property enabled by default.

• Solr destination enhancements - The destination includes the following enhancements:
  • The destination now includes an Ignore Optional Fields property that allows ignoring null values in optional fields when writing records.
  • The destination allows you to configure Wait Flush, Wait Searcher, and Soft Commit properties to tune write performance.

• Edge pipelines now support the following stages:
  • Dev Raw Data Source origin
  • Kafka Producer destination
• Edge pipelines now support the following functions:
  • emptyList()
  • emptyMap()
  • isEmptyMap()
  • isEmptyList()
  • length()
  • record:attribute()
  • record:attributeOrDefault()
  • size()
• When you start SDC Edge, you can now change the default log directory.

• HTTP Client origin enhancement - You can now configure the origin to use the Link in Response Field pagination type. After processing the current page, this pagination type uses a field in the response body to access the next page.
• HTTP Server origin enhancement - You can now use the origin to process the contents of authorized HTTP PUT requests.
• Kinesis Consumer origin enhancement - You can now define tags to apply to the DynamoDB lease table that the origin creates to store offsets.
• MQTT Subscriber origin enhancement - The origin now includes a TOPIC_HEADER_NAME record header attribute that includes the topic information for each record.
• MongoDB origin enhancement - The origin now generates a no-more-data event when it has processed all available documents and the configured batch wait time has elapsed.
• Oracle CDC Client origin enhancement - You can now specify the tables to process by using SQL-like syntax in table inclusion patterns and exclusion patterns.
• Salesforce origin enhancements - The origin includes the following enhancements:
  • The origin can now subscribe to Salesforce platform events.
  • You can now configure the origin to use Salesforce PK Chunking.
  • When necessary, you can disable query validation.
  • You can now use Mutual Authentication to connect to Salesforce.

• New Field Replacer processor - A new processor that replaces values in fields with nulls or with new values.

  The Field Replacer processor replaces the Value Replacer processor which has been deprecated. The Field Replacer processor lets you define more complex conditions to replace values. For example, the Field Replacer can replace values which fall within a specified range. The Value Replacer cannot replace values that fall within a specified range.

  StreamSets recommends that you update Value Replacer pipelines as soon as possible.

• New Postgres Metadata processor - A new processor that determines when changes in data structure occur and creates and alters PostgreSQL tables accordingly. Use as part of the Drift Synchronization Solution for Postgres in development or testing environments only.
• Aggregator processor enhancements - The processor includes the following enhancements:
  • Event records now include the results of the aggregation.
  • You can now specify the root field for event records. You can use a String or Map root field. Upgraded pipelines retain the previous behavior, writing aggregation data to a String root field.
• JDBC Lookup processor enhancement - The processor includes the following enhancements:
  • You can now configure a Missing Values Behavior property that defines processor behavior when a lookup returns no value. Upgraded pipelines continue to send records with no return value to error.
  • You can now enable the Retry on Cache Miss property so that the processor retries lookups for known missing values. By default, the processor always returns the default value for known missing values to avoid unnecessary lookups.
• Kudu Lookup processor enhancement - The processor no longer requires that you add a primary key column to the Key Columns Mapping. However, adding only non-primary keys can slow the performance of the lookup.
• Salesforce Lookup processor enhancement - You can now use Mutual Authentication to connect to Salesforce.

• New Aerospike destination - A new destination that writes data to Aerospike.
• New Named Pipe destination- A new destination that writes data to a UNIX named pipe.
• Einstein Analytics destination enhancements - The destination includes the following enhancements:
  • You can specify the name of the edgemart container that contains the dataset.
  • You can define the operation to perform: Append, Delete, Overwrite, or Upsert.
  • You can now use Mutual Authentication to connect to Salesforce.
• Elasticsearch destination enhancement - You can now configure the destination to merge data which performs an update with doc_as_upsert.
• Salesforce destination enhancement - The destination includes the following enhancements:
  • The destination can now publish Salesforce platform events.
  • You can now use Mutual Authentication to connect to Salesforce.

• Log data format enhancement - Data Collector can now process data using the following log format types:
  • Common Event Format (CEF)
  • Log Event Extended Format (LEEF)

• Error record functions - This release includes the following new function:
  • record:errorStackTrace() - Returns the error stack trace for the record.
• Time functions - This release includes the following new functions:
  • time:dateTimeZoneOffset() - Returns the time zone offset in milliseconds for the specified date and time zone.
  • time:timeZoneOffset() - Returns the time zone offset in milliseconds for the specified time zone.
• Miscellaneous functions - This release includes the following changed and new functions:
  • runtime:loadResource() - This function has been changed to trim any leading or trailing whitespace characters from the file before returning the value in the file. Previously, the function did not trim white space characters - you had to avoid including unnecessary characters in the file.
  • runtime:loadResourceRaw() - New function that returns the value in the specified file, including any leading or trailing whitespace characters in the file.

• Apache Kudu 1.6
• Cloudera 5.13 distribution of Apache Kafka 2.1
• Cloudera 5.14 distribution of Apache Kafka 2.1
• Cloudera CDH 5.14 distribution of Hadoop
• Kinetica 6.1

• Data Collector classpath validation - Data Collector now performs a classpath health check upon starting up. The results of the health check are written to the Data Collector log. When necessary, you can configure Data Collector to skip the health check or to stop upon errors.
• Support bundle Data Collector property - You can configure a property in the Data Collector configuration file to have Data Collector automatically upload support bundles when problems occur. The property is disabled by default.
• Redirect registered Data Collector user logins property - You can enable a property in the Control Hub configuration file, dpm.properties, to redirect Data Collector user logins to Control Hub using the HTML meta refresh method.
• Runtime properties enhancement - You can now use environment variables in runtime properties.

• Java requirement - Data Collector now supports both Oracle Java 8 and OpenJDK 8.
• RPM packages - StreamSets now provides the following Data Collector RPM packages:
  • EL6 - Use to install Data Collector on CentOS 6, Oracle Linux 6, or Red Hat Enterprise Linux 6
  • EL7 - Use to install Data Collector on CentOS 7, Oracle Linux 7, or Red Hat Enterprise Linux 7.

  Previously, StreamSets provided a single RPM package used to install Data Collector on any of these operating systems.

• New Amazon SQS Consumer origin - An origin that reads messages from Amazon Simple Queue Service (SQS). Can create multiple threads to enable parallel processing in a multithreaded pipeline.
• New Google Cloud Storage origin - An origin that reads fully written objects in Google Cloud Storage.
• New MapR DB CDC origin - An origin that reads changed MapR DB data that has been written to MapR Streams. Can create multiple threads to enable parallel processing in a multithreaded pipeline.
• New MapR Multitopic Streams Consumer origin - An origin that reads messages from multiple MapR Streams topics. It can create multiple threads to enable parallel processing in a multithreaded pipeline.
• New UDP Multithreaded Source origin - The origin listens for UDP messages on one or more ports and queues incoming packets on an intermediate queue for processing. It can create multiple threads to enable parallel processing in a multithreaded pipeline.
• New WebSocket Client origin - An origin that reads data from a WebSocket server endpoint.
• New Windows Event Log origin - An origin that reads data from Microsoft Windows event logs. You can use this origin only in pipelines configured for edge execution mode.
• New Sensor Reader development origin - A development origin that generates sample atmospheric data for edge pipelines.
• Amazon S3 origin enhancements:
  • The origin now produces no-more-data events and includes a new socket timeout property.
  • You can now specify the number of times the origin retries a query. The default is three.
• Directory origin enhancement - The origin can now use multiple threads to perform parallel processing of files.
• HTTP Client origin enhancement - The origin can now log request and response data to the Data Collector log.
• JDBC Multitable Consumer origin enhancements:
  • The origin can now use non-incremental processing for tables with no primary key or offset column.
  • You can now specify an Init Query to be executed after establishing a connection to the database, before performing other tasks. This can be used, for example, to modify session attributes.
  • A new Queries Per Second property determines how many queries can be run every second.

    This property replaces the Query Interval property. For information about possible upgrade impact, see JDBC Multitable Consumer Query Interval Change.

• JDBC Query Consumer origin enhancements:
  • You can now specify an Init Query to be executed after establishing a connection to the database, before performing other tasks. This can be used, for example, to modify session attributes.
  • The Microsoft SQL Server CDC functionality in the JDBC Query Consumer origin is now deprecated and will be removed from the origin in a future release. For upgrade information, see Update JDBC Query Consumer Pipelines used for SQL Server CDC Data.
• Kafka Multitopic Consumer origin enhancement - The origin is now available in the following stage libraries, in addition to the Apache Kafka 0.10 stage library:

  • Apache Kafka 0.9

  • CDH Kafka 2.0 (0.9.0) and 2.1 (0.9.0)

  • HDP 2.5 and 2.6

• Kinesis Consumer origin enhancement - You can now specify the number of times the origin retries a query. The default is three.
• Oracle CDC Client origin enhancements:
  • When using SCNs for the initial change, the origin now treats the specified SCN as a starting point rather than looking for an exact match.
  • The origin now passes raw data to the pipeline as a byte array.
  • The origin can now include unparsed strings from the parsed SQL query for unsupported data types in records.
  • The origin now uses local buffering instead of Oracle LogMiner buffering by default. Upgraded pipelines require no changes.
  • The origin now supports reading the Timestamp with Timezone data type. When reading Timestamp with Timezone data, the origin includes the offset with the datetime data in the Data Collector Zoned Datetime data type. It does not include the time zone ID.
• SQL Server CDC Client origin enhancements - You can now perform the following tasks with the SQL Server CDC Client origin:
  • Process CDC tables that appear after the pipeline starts.
  • Check for schema changes and generate events when they are found.

  • In addition, a new Capture Instance Name property replaces the Schema and Table Name Pattern properties from earlier releases.

    You can simply use the schema name and table name pattern for the capture instance name. Or, you can specify the schema name and a capture instance name pattern, which allows you to specify specific CDC tables to process when you have multiple CDC tables for a single data table.

    Upgraded pipelines require no changes.

• UDP Source origin enhancement - The Enable Multithreading property that enabled using multiple epoll receiver threads is now named Use Native Transports (epoll).

• New Aggregator processor - A processor that aggregates data within a window of time. Displays the results in Monitor mode and can write the results to events.
• New Delay processor - A processor that can delay processing a batch of records for a specified amount of time.
• Field Type Converter processor enhancement - You can now convert strings to the Zoned Datetime data type, and vice versa. You can also specify the format to use.
• Hive Metadata processor enhancement - You can now configure additional JDBC configuration properties to pass to the JDBC driver.
• HTTP Client processor enhancements:
  • The processor can now log request and response data to the Data Collector log.
  • The Rate Limit now defines the minimum amount of time between requests in milliseconds. Previously, it defined the time between requests in seconds. Upgraded pipelines require no changes.
• JDBC Lookup and JDBC Tee processor enhancements - You can now specify an Init Query to be executed after establishing a connection to the database, before performing other tasks. This can be used, for example, to modify session attributes.
• Kudu Lookup processor enhancement - The Cache Kudu Table property is now named Enable Table Caching. The Maximum Entries to Cache Table Objects property is now named Maximum Table Entries to Cache.
• Salesforce Lookup processor enhancement - You can use a new Retrieve lookup mode to look up data for a set of records instead of record-by-record. The mode provided in previous releases is now named SOQL Query. Upgraded pipelines require no changes.

• New Google Cloud Storage destination - A new destination that writes data to objects in Google Cloud Storage. The destination can generate events for use as dataflow triggers.
• New KineticaDB destination - A new destination that writes data to a Kinetica table.
• Amazon S3 destination enhancement - You can now specify the number of times the destination retries a query. The default is three.
• Hive Metastore destination enhancement - You can now configure additional JDBC configuration properties to pass to the JDBC driver.
• HTTP Client destination enhancements:
  • The destination can now log request and response data to the Data Collector log.
  • You can now use the HTTP Client destination to write Avro, Delimited, and Protobuf data in addition to the previous data formats.
• JDBC Producer destination enhancement - You can now specify an Init Query to be executed after establishing a connection to the database, before performing other tasks. This can be used, for example, to modify session attributes.
• Kudu destination enhancement - If the destination receives a change data capture log from the following source systems, you now must specify the source system in the Change Log Format property so that the destination can determine the format of the log: Microsoft SQL Server, Oracle CDC Client, MySQL Binary Log, or MongoDB Oplog.
• MapR DB JSON destination enhancement - The destination now supports writing to MapR DB based on the CRUD operation in record header attributes and the Insert API and Set API properties.
• MongoDB destination enhancements - With this release, the Upsert operation is no longer supported by the destination. Instead, the destination includes the following enhancements:
  • Support for the Replace and Update operations.
  • Support for an Upsert flag that, when enabled, is used with both the Replace and Update operations.

  For information about upgrading existing upsert pipelines, see Update MongoDB Destination Upsert Pipelines.

• Redis destination enhancement - The destination now supports processing data using CRUD operations stored in record header attributes.
• Salesforce destination enhancement - When using the Salesforce Bulk API to update, insert, or upsert data, you can now use a colon (:) or period (.) as a field separator when defining the Salesforce field to map the Data Collector field to. For example, Parent__r:External_Id__c or Parent__r.External_Id__c are both valid Salesforce fields.
• Wave Analytics destination rename - With this release, the Wave Analytics destination is now named the Einstein Analytics destination, following the recent Salesforce rebranding. All of the properties and functionality of the destination remain the same.

• Hive Query executor enhancement - You can now configure additional JDBC configuration properties to pass to the JDBC driver.
• JDBC Query executor enhancement - You can now specify an Init Query to be executed after establishing a connection to the database, before performing other tasks. This can be used, for example, to modify session attributes.

For information about upgrading a version of Data Collector with Cloudera Navigator integration enabled, see Disable Cloudera Navigator Integration.

• CyberArk - Data Collector now provides a credential store implementation for CyberArk Application Identity Manager. You can define the credentials required by external systems - user names or passwords - in CyberArk. Then you use credential expression language functions in JDBC stage properties to retrieve those values, instead of directly entering credential values in stage properties.
• Supported stages - You can now use the credential functions in all stages that require you to enter sensitive information. Previously, you could only use the credential functions in JDBC stages.

• StreamSets Control Hub - With this release, we have created a new product called StreamSets Control Hub^TM that includes a number of new cloud-based dataflow design, deployment, and scale-up features. Since this release is now our core service for controlling dataflows, we have renamed the StreamSets cloud experience from "Dataflow Performance Manager (DPM)" to "StreamSets Control Hub".

  DPM now refers to the performance management functions that reside in the cloud such as live metrics and data SLAs. Customers who have purchased the StreamSets Enterprise Edition will gain access to all Control Hub functionality and continue to have access to all DPM functionality as before.

  To understand the end-to-end StreamSets Data Operations Platform and how the products fit together, visit https://streamsets.com/products/.

• Aggregated statistics - When working with Control Hub, you can now configure a pipeline to write aggregated statistics to MapR Streams.

• New NetFlow 9 support - Data Collector now supports processing NetFlow 9 template-based messages. Stages that previously processed NetFlow 5 data can now process NetFlow 9 data as well.
• Datagram data format enhancement - The Datagram Data Format property is now named the Datagram Packet Format.
• Delimited data format enhancement - Data Collector can now process data using the Postgres CSV and Postgres Text delimited format types.

• New field path expressions - You can use field path expressions in certain stages to specify the fields to use in an expression.
• New field functions - You can use the following new field functions in field path expressions:
  • f:attribute() - Returns the value of the specified attribute.
  • f:path() - Returns the path of a field.
  • f:type() - Returns the data type of a field.
  • f:value() - Returns the value of a field.
• New string functions - The release includes the following new functions:
  • str:isNullOrEmpty() - Returns true or false based on whether a string is null or is the empty string.
  • str:splitKV() - Splits key-value pairs in a string into a map of string values.

• New stage libraries - This release includes the following new stage libraries:
  • Apache Kafka 1.0
  • Apache Kafka 0.11
  • Apache Kudu 1.5
  • Cloudera CDH 5.13
  • Cloudera Kafka 3.0.0 (0.11.0)
  • Hortonworks 2.6.1, including Hive 1.2
  • Hortonworks 2.6.2, including Hive 1.2 and 2.0
  • MapR version 6.0 (MEP 4)
  • MapR Spark 2.1 (MEP 3)
• Legacy stage libraries - Stage libraries that are more than two years old are no longer included with Data Collector. Though not recommended, you can still download and install the older stage libraries as custom stage libraries.

  If you have pipelines that use these legacy stage libraries, you will need to update the pipelines to use a more current stage library or install the legacy stage library manually, For more information see Update Pipelines using Legacy Stage Libraries.

• Statistics stage library enhancement - The statistics stage library is now included in the core Data Collector installation.

• New data type - Data Collector now supports the Zoned Datetime data type.
• New Data Collector metrics - JVM metrics have been renamed Data Collector Metrics and now include general Data Collector metrics in addition to JVM metrics. The JVM Metrics menu item has also been renamed SDC Metrics.
• Pipeline error records - You can now write error records to Google Pub/Sub, Google Cloud Storage, or an MQTT broker.
• Snapshot enhancements:
  • Standalone pipelines can now automatically take a failure snapshot when the pipeline fails due to a data-related exception.
  • You can now download snapshots through the UI and the REST API.
• Time zone enhancement - Time zones have been organized and updated to use JDK 8 names. This should make it easier to select time zones in stage properties. In the rare case that your pipeline uses a format not supported by JDK 8, edit the pipeline to select a compatible time zone.

• New Kafka Multitopic Consumer origin - A new origin that reads messages from multiple Kafka topics. Creates multiple threads to enable parallel processing in a multithreaded pipeline.
• Kinesis Consumer origin enhancement - You can now configure the origin to start reading messages from a specified timestamp.

• New Google BigQuery destination - A new destination that streams data into Google BigQuery.

• Amazon S3 origin enhancement - You can now specify a Connection Timeout advanced property.
• JDBC Multitable Consumer origin enhancement - You can now use the origin to read from views in addition to tables.
• OPC UA Client origin enhancement - You can now configure channel properties, such as the maximum chunk or message size.
• Oracle CDC Client origin enhancement - You can now configure a JDBC Fetch Size property to determine the minimum number of records that the origin waits for before passing data to the pipeline. When writing to the destination is slow, use the default of 1 record to improve performance. Previously, the origin used the Oracle JDBC driver default of 10 records.

• New MapR FS File Metadata executor - The new executor can change file metadata, create an empty file, or remove a file or directory in MapR each time it receives an event.

• Jython 2.7.1

• New Azure Event Hub Consumer origin - A multithreaded origin that reads data from Microsoft Azure Event Hub.
• OPC UA Client origin enhancement - You can now specify node information in a file. Or have the origin browse for nodes to use based on a specified root node.

• New Schema Generator processor - A processor that generates a schema for each record and writes the schema to a record header attribute.

• New Azure Event Hub Producer destination - A destination that writes data to Microsoft Azure Event Hub.
• New Azure IoT Hub Producer destination - A destination that writes data to Microsoft Azure IoT Hub.

• MapR prerequisites - You can now run the setup-mapr command in interactive or non-interactive mode. In interactive mode, the command prompts you for the MapR version and home directory. In non-interactive mode, you define the MapR version and home directory in environment variables before running the command.

• Hortonworks version 2.6 distribution of Apache Hadoop

• Cloudera distribution of Spark 2.1
• MapR distribution of Spark 2.1

• New buffer size configuration - You can now use a new parser.limit configuration property to increase the Data Collector parser buffer size. The parser buffer is used by the origin to process many data formats, including Delimited, JSON, and XML. The parser buffer size limits the size of the records that origins can process. The Data Collector parser buffer size is 1048576 bytes by default.

• Parquet support - You can now use the Drift Synchronization Solution for Hive to generate Parquet files. Previously, the Data Synchronization Solution supported only Avro data. This enhancement includes the following updates:
  • Hive Metadata processor data format property - Use the new data format property to indicate the data format to use.
  • Parquet support in the Hive Metastore destination - The destination can now create and update Parquet tables in Hive. The destination no longer includes a data format property since that information is now configured in the Hive Metadata processor.

• Origins for multithreaded pipelines - You can now use the following origins to create multithreaded pipelines:

  • CoAP Server origin

  • TCP Server origin

• Multithreaded origin icons - The icons for multithreaded origins now include the following multithreaded indicator:

  For example, here’s the updated Elasticsearch origin icon:

  

• New executors - You can now use the following executors to perform tasks upon receiving an event:
  • Email executor
  • Shell executor

• Pipeline statistics - You can now configure a pipeline to write statistics directly to DPM. Write statistics directly to DPM when you run a job for the pipeline on a single Data Collector.

  When you run a job on multiple Data Collectors, a remote pipeline instance runs on each of the Data Collectors. To view aggregated statistics for the job within DPM, you must configure the pipeline to write the statistics to a Kafka cluster, Amazon Kinesis Streams, or SDC RPC.

• Update published pipelines - When you update a published pipeline, Data Collector now displays a red asterisk next to the pipeline name to indicate that the pipeline has been updated since it was last published.

• New CoAP Server origin - An origin that listens on a CoAP endpoint and processes the contents of all authorized CoAP requests. The origin performs parallel processing and can generate multithreaded pipelines.
• New TCP Server origin - An origin that listens at the specified ports, establishes TCP sessions with clients that initiate TCP connections, and then processes the incoming data. The origin can process NetFlow, syslog, and most Data Collector data formats as separated records. You can configure custom acknowledgement messages and use a new batchSize variable, as well as other expressions, in the messages.
• SFTP/FTP Client origin enhancement - You can now specify the first file to process. This enables you to skip processing files with earlier timestamps.

• Groovy, JavaScript, and Jython Evaluator processor enhancements:
  • You can now include some methods of the sdcFunctions scripting object in the initialization and destroy scripts for the processors.
  • You can now use runtime parameters in the code developed for a Groovy Evaluator processor.
• Hive Metadata processor enhancements:
  • The Hive Metadata processor can now process Parquet data as part of the Drift Synchronization Solution for Hive.
  • You can now specify the data format to use: Avro or Parquet.
  • You can now configure an expression that defines comments for generated columns.
• JDBC Lookup processor enhancements:
  • The JDBC Lookup processor can now return multiple values. You can now configure the lookup to return the first value or to return all matches as separate records.
  • When you monitor a pipeline that includes the JDBC Lookup processor, you can now view stage statistics about the number of queries the processor makes and the average time of the queries.
• Spark Evaluator enhancement - The Spark Evaluator now supports Spark 2.x.

• New CoAP Client destination - A destination that writes to a CoAP endpoint.
• Hive Metastore destination enhancements:
  • The destination can now create and update Parquet tables in Hive.
  • Also, the data format property has been removed. You now specify the data format in the Hive Metadata processor.

    Since the Hive Metastore previously supported only Avro data, there is no upgrade impact.

• Kudu destination enhancement - You can use the new Mutation Buffer Space property to set the buffer size that the Kudu client uses to write each batch.

• New Email executor - Use to send custom emails upon receiving an event. For a case study, see Case Study: Sending Email.

  • New Shell executor - Use to execute shell scripts upon receiving an event.

  • JDBC Query executor enhancement - A new Batch Commit property allows the executor to commit to the database after each batch. Previously, the executor did not call commits by default.

    For new pipelines, the property is enabled by default. For upgraded pipelines, the property is disabled to prevent changes in pipeline behavior.

  • Spark executor enhancement - The executor now supports Spark 2.x.

• Offset management - Both the REST API and command line interface can now retrieve the last-saved offset for a pipeline and set the offset for a pipeline when it is not running. Use these commands to implement pipeline failover using an external storage system. Otherwise, pipeline offsets are managed by Data Collector and there is no need to update the offsets.

• vault:read enhancement - The vault:read function now supports returning the value for a key nested in a map.

• Support bundles - You can now use Data Collector to generate a support bundle. A support bundle is a ZIP file that includes Data Collector logs, environment and configuration information, pipeline JSON files, resource files, and pipeline snapshots.

  You upload the generated file to the StreamSets support team so that we can use the information to troubleshoot your support tickets.

• TLS property enhancements - Stages that support SSL/TLS now provide the following enhanced set of properties that enable more specific configuration:

  • Keystore and truststore type - You can now choose between Java Keystore (JKS) and PKCS #12 (p-12). Previously, Data Collector only supported JKS.
  • Transport protocols - You can now specify the transport protocols that you want to allow. By default, Data Collector allows only TLSv1.2.
  • Cipher suites - You can now specify the cipher suites to allow. Data Collector provides a modern set of default cipher suites. Previously, Data Collector always allowed the default cipher suites for the JRE.

  To avoid upgrade impact, all SSL/TLS/HTTPS properties in existing pipelines are preserved during upgrade.

• Cluster mode enhancement - Cluster streaming mode now supports Spark 2.x. For information about using Spark 2.x stages with cluster mode, see Cluster Pipeline Limitations.

• Precondition enhancement - Stages with user-defined preconditions now process all preconditions before passing a record to error handling. This allows error records to include all precondition failures in the error message.
• Pipeline import/export enhancement - When you export multiple pipelines, Data Collector now includes all pipelines in a single zip file. You can also import multiple pipelines from a single zip file.

• Register Data Collectors with DPM - If Data Collector uses file-based authentication and if you register the Data Collector from the Data Collector UI, you can now create DPM user accounts and groups during the registration process.
• Aggregated statistics for DPM - When working with DPM, you can now configure a pipeline to write aggregated statistics to SDC RPC. Write statistics to SDC RPC for development purposes only. For a production environment, use a Kafka cluster or Amazon Kinesis Streams to aggregate statistics.

• Dev SDC RPC with Buffering origin - A new development stage that receives records from an SDC RPC destination, temporarily buffering the records to disk before passing the records to the next stage in the pipeline. Use as the origin in an SDC RPC destination pipeline.
• Amazon S3 origin enhancement - You can configure a new File Pool Size property to determine the maximum number of files that the origin stores in memory for processing after loading and sorting all files present on S3.

• New stage libraries - This release supports the following new stage libraries:
  • Kudu versions 1.1 and 1.2

  • Cloudera CDH version 5.10 distribution of Hadoop

  • Cloudera version 5.10 distribution of Apache Kafka 2.1
• Install external libraries using the Data Collector user interface - You can now use the Data Collector user interface to install external libraries to make them available to stages. For example, you can install JDBC drivers for stages that use JDBC connections. Or, you can install external libraries to call external Java code from the Groovy, Java, and Jython Evaluator processors.
• Custom header enhancement - You can now use HTML in the ui.header.title configuration property to configure a custom header for the Data Collector UI. This allows you to specify the look and feel for any text that you use, and to include small images in the header.
• Groovy enhancement - You can configure the processor to use the invokedynamic bytecode instruction.
• Pipeline renaming - You can now rename a pipeline by clicking directly on the pipeline name when editing the pipeline, in addition to editing the Title general pipeline property.

The new multithreaded framework includes the following changes:

• HTTP Server origin - Listens on an HTTP endpoint and processes the contents of all authorized HTTP POST requests. Use the HTTP Server origin to receive high volumes of HTTP POST requests using multiple threads.

• Enhanced Dev Data Generator origin - Can create multiple threads for testing multithreaded pipelines.

• Enhanced runtime statistics - Monitoring a pipeline displays aggregated runtime statistics for all threads in the pipeline. You can also view the number of runners, i.e. threads and pipeline instances, being used.

CDC-enabled origins:

• The MongoDB Oplog and Salesforce origins are now enabled for processing changed data by including the CRUD operation type in the sdc.operation.type record header attribute.

• Though previously CDC-enabled, the Oracle CDC Client and JDBC Query Consumer for Microsoft SQL Server now include CRUD operation type in the sdc.operation.type record header attribute.

  Previous operation type header attributes are still supported for backward-compatibility.

CRUD-enabled stages:

• The JDBC Tee processor and JDBC Producer can now process changed data based on CRUD operations in record headers. The stages also include a default operation and unsupported operation handling.

• The MongoDB and Elasticsearch destinations now look for the CRUD operation in the sdc.operation.type record header attribute. The Elasticsearch destination includes a default operation and unsupported operation handling.

• Groups for file-based authentication - If you use file-based authentication, you can now create groups of users when multiple users use Data Collector. You configure groups in the associated realm.properties file located in the Data Collector configuration directory, $SDC_CONF.

  If you use file-based authentication, you can also now view all user accounts granted access to the Data Collector, including the roles and groups assigned to each user.

• LDAP authentication enhancements - You can now configure Data Collector to use StartTLS to make secure connections to an LDAP server. You can also configure the userFilter property to define the LDAP user attribute used to log in to Data Collector. For example, a username, uid, or email address.

• Proxy configuration for outbound requests - You can now configure Data Collector to use an authenticated HTTP proxy for outbound requests to Dataflow Performance Manager (DPM).

• Java garbage collector logging - Data Collector now enables logging for the Java garbage collector by default. Logs are written to $SDC_LOG/gc.log. You can disable the logging if needed.

• Heap dump for out of memory errors - Data Collector now produces a heap dump file by default if it encounters an out of memory error. You can configure the location of the heap dump file or you can disable this default behavior.
• Modifying the log level - You can now use the Data Collector UI to modify the log level to display messages at another severity level.

• Pipeline renaming - You can now rename pipelines by editing the Title general pipeline property.

• Field attributes - Data Collector now supports field-level attributes. Use the Expression Evaluator to add field attributes.

• New HTTP Server origin - A multithreaded origin that listens on an HTTP endpoint and processes the contents of all authorized HTTP POST requests. Use the HTTP Server origin to read high volumes of HTTP POST requests using multiple threads.

• New HTTP to Kafka origin - Listens on a HTTP endpoint and writes the contents of all authorized HTTP POST requests directly to Kafka. Use to read high volumes of HTTP POST requests and write them to Kafka.

• New MapR DB JSON origin - Reads JSON documents from MapR DB JSON tables.

• New MongoDB Oplog origin - Reads entries from a MongoDB Oplog. Use to process change information for data or database operations.

• Directory origin enhancement - You can use regular expressions in addition to glob patterns to define the file name pattern to process files.

• HTTP Client origin enhancement - You can now configure the origin to use the OAuth 2 protocol to connect to an HTTP service.

• JDBC Query Consumer origin enhancements - The JDBC Consumer origin has been renamed to the JDBC Query Consumer origin. The origin functions the same as in previous releases. It reads database data using a user-defined SQL query through a JDBC connection. You can also now configure the origin to enable auto-commit mode for the JDBC connection and to disable validation of the SQL query.
• MongoDB origin enhancements - You can now use a nested field as the offset field. The origin supports reading the MongoDB BSON timestamp for MongoDB versions 2.6 and later. And you can configure the origin to connect to a single MongoDB server or node.

• Field Type Converter processor enhancement - You can now configure the processor to convert timestamp data in a long field to a String data type. Previously, you had to use one Field Type Converter processor to convert the long field to a datetime, and then use another processor to convert the datetime field to a string.

• HTTP Client processor enhancements - You can now configure the processor to use the OAuth 2 protocol to connect to an HTTP service. You can also configure a rate limit for the processor, which defines the maximum number of requests to make per second.

• JDBC Lookup processor enhancements - You can now configure the processor to enable auto-commit mode for the JDBC connection. You can also configure the processor to use a default value if the database does not return a lookup value for a column.

• Salesforce Lookup processor enhancement - You can now configure the processor to use a default value if Salesforce does not return a lookup value for a field.

• XML Parser enhancement - A new Multiple Values Behavior property allows you to specify the behavior when you define a delimiter element and the document includes more than one value: Return the first value as a record, return one record with a list field for each value, or return all values as records.

• New MapR DB JSON destination - Writes data as JSON documents to MapR DB JSON tables.

• Azure Data Lake Store destination enhancement - You can now use the destination in cluster batch pipelines. You can also process binary and protobuf data, use record header attributes to write records to files and roll files, and configure a file suffix and the maximum number of records that can be written to a file.

• Elasticsearch destination enhancement - The destination now uses the Elasticsearch HTTP API. With this API, the Elasticsearch version 5 stage library is compatible with all versions of Elasticsearch. Earlier stage library versions have been removed. Elasticsearch is no longer supported on Java 7. You’ll need to verify that Java 8 is installed on the Data Collector machine and remove this stage from the blacklist property in $SDC_CONF/sdc.properties before you can use it.

  You can also now configure the destination to perform any of the following CRUD operations: create, update, delete, or index.

• Hive Metastore destination enhancement - New table events now include information about columns and partitions in the table.

• Hadoop FS, Local FS, and MapR FS destination enhancement - The destinations now support recovery after an unexpected stop of the pipeline by renaming temporary files when the pipeline restarts.

• Redis destination enhancement - You can now configure a timeout for each key that the destination writes to Redis.

• Hive Query executor enhancements:

  • The executor can now execute multiple queries for each event that it receives.
  • It can also generate event records each time it processes a query.

• JDBC Query executor enhancement - You can now configure the executor to enable auto-commit mode for the JDBC connection.

• Whole File enhancement - You can now specify a transfer rate to help control the resources used to process whole files. You can specify the rate limit in all origins that process whole files.

• New pipeline functions - You can use the following new pipeline functions to return pipeline information:
  • pipeline:id() - Returns the pipeline ID, a UUID that is automatically generated and used by Data Collector to identify the pipeline.
    Note: The existing pipeline:name() function now returns the pipeline ID instead of the pipeline name since pipeline ID is the correct way to identify a pipeline.

  • pipeline:title() - Returns the pipeline title or name.

• New record functions - You can use the following new record functions to work with field attributes:

  • record:fieldAttribute (<field path>, <attribute name>) - Returns the value of the specified field attribute.
  • record:fieldAttributeOrDefault (<field path>, <attribute name>, <default value>) - Returns the value of the specified field attribute. Returns the default value if the attribute does not exist or contains no value.
• New string functions - You can use the following new string functions to transform string data:

  • str:urlEncode (<string>, <encoding>) - Returns a URL encoded string from a decoded string using the specified encoding format.
  • str:urlDecode (<string>, <encoding>) - Returns a decoded string from a URL encoded string using the specified encoding format.
• New time functions - You can use the following new time functions to transform datetime data:

  • time:dateTimeToMilliseconds (<Date object>) - Converts a Date object to an epoch or UNIX time in milliseconds.
  • time:extractDateFromString(<string>, <format string>) - Extracts a Date object from a String, based on the specified date format.
  • time:extractStringFromDateTZ (<Date object>, <timezone>, <format string>) - Extracts a string value from a Date object based on the specified date format and time zone.
• New and enhanced miscellaneous functions - You can use the following new and enhanced miscellaneous functions:

  • offset:column(<position>) - Returns the value of the positioned offset column for the current table. Available only in the additional offset column conditions of the JDBC Multitable Consumer origin.
  • every function - You can now use the function with the hh() datetime variable in directory templates. This allows you to create directories based on the specified interval for hours.

• New Field Zip processor - Merges two List fields or two List-Map fields in the same record.
• New Salesforce Lookup processor - Performs lookups in a Salesforce object and passes the lookup values to fields. Use the Salesforce Lookup to enrich records with additional data.
• Value Replacer enhancement - You can now replace field values with nulls using a condition.

• Whole file support in the Azure Data Lake Store destination - You can now use the whole file data format to stream whole files to Azure Data Lake Store.

• Installation requirements:
  • Java requirement - Oracle Java 7 is supported but now deprecated. Oracle announced the end of public updates for Java 7 in April 2015. StreamSets recommends migrating to Java 8, as Java 7 support will be removed in a future Data Collector release.
  • File descriptors requirement - Data Collector now requires a minimum of 32,768 open file descriptors.

• Core installation includes the basic stage library only - The core RPM and tarball installations now include the basic stage library only, to allow Data Collector to use less disk space. Install additional stage libraries using the Package Manager for tarball installations or the command line for RPM and tarball installations.

  Previously, the core installation also included the Groovy, Jython, and statistics stage libraries.

• New stage libraries. Data Collector now supports the following stage libraries:
  • Apache Kudu version 1.0.x - Earlier Kudu versions are no longer supported.
  • Cloudera CDH version 5.9 distribution of Apache Hadoop.
  • Cloudera version 5.9 distribution of Apache Kafka 2.0.
  • Elasticsearch version 5.0.x.
  • Google Cloud Bigtable.
  • Hortonworks HDP version 2.5 distribution of Apache Hadoop.
  • MySQL Binary Log.
  • Salesforce.
• LDAP authentication - If you use LDAP authentication, you can now configure Data Collector to connect to multiple LDAP servers. You can also configure Data Collector to support an LDAP deployment where members are defined by uid or by full DN.
• Java garbage collector - Data Collector now uses the Concurrent Mark Sweep (CMS) garbage collector by default. You can configure Data Collector to use a different garbage collector by modifying Java configuration options in the Data Collector environment configuration file.
• Environment variables for Java configuration options. Data Collector now uses three environment variables to define Java configuration options:
  • SDC_JAVA_OPTS - Includes configuration options for all Java versions. SDC_JAVA7_OPTS - Includes configuration options used only when Data Collector is running Java 7.
  • SDC_JAVA8_OPTS - Includes configuration options used only when Data Collector is running Java 8.
• New time zone property - You can configure the Data Collector UI to use UTC, the browser time zone, or the Data Collector time zone. The time zone property affects how dates and times display in the UI. The default is the browser time zone.

• New MySQL Binary Log origin - Reads MySQL binary logs to generate records with change data capture information.
• New Salesforce origin - Reads data from Salesforce. The origin can execute a SOQL query to read existing data from Salesforce. The origin can also subscribe to the Force.com Streaming API to receive notifications for changes to Salesforce data.
• Directory origin enhancement - You can configure the Directory origin to read files from all subdirectories when using the last-modified timestamp for the read order.
• JDBC Query Consumer and Oracle CDC Client origin enhancement - You can now configure the transaction isolation level that the JDBC Query Consumer and Oracle CDC Client origins use to connect to the database. Previously, the origins used the default transaction isolation level configured for the database.

• New Spark Evaluator processor - Processes data based on a Spark application that you develop. Use the Spark Evaluator processor to develop a Spark application that performs custom processing within a pipeline.
• Field Flattener processor enhancements - In addition to flattening the entire record, you can also now use the Field Flattener processor to flatten specific list or map fields in the record.
• Field Type Converter processor enhancements - You can now use the Field Type Converter processor to change the scale of a decimal field. Or, if you convert a field with another data type to the Decimal data type, you can configure the scale to use in the conversion.
• Field Pivoter processor enhancements - The List Pivoter processor has been renamed to the Field Pivoter processor. You can now use the processor to pivot data in a list, map, or list-map field. You can also use the processor to save the field name of the first-level item in the pivoted field.
• JDBC Lookup and JDBC Tee processor enhancement - You can now configure the transaction isolation level that the JDBC Lookup and JDBC Tee processors use to connect to the database. Previously, the origins used the default transaction isolation level configured for the database.
• Scripting processor enhancements - The Groovy Evaluator, JavaScript Evaluator, and Jython Evaluator processors can generate event records and work with record header attributes. The sample scripts now include examples of both and a new tip for generating unique record IDs.
• XML Flattener processor enhancement - You can now configure the XML Flattener processor to write the flattened data to a new output field. Previously, the processor wrote the flattened data to the same field.
• XML Parser processor enhancement. You can now generate records from XML documents using simplified XPath expressions. This enables reading records from deeper within XML documents.

• New Azure Data Lake Store destination - Writes data to Microsoft Azure Data Lake Store.
• New Google Bigtable destination - Writes data to Google Cloud Bigtable.
• New Salesforce destination - Writes data to Salesforce. New Wave Analytics destination. Writes data to Salesforce Wave Analytics. The destination creates a dataset with external data.
• Amazon S3 destination change - The AWS KMS Key ID property has been renamed AWS KMS Key ARN. Data Collector upgrades existing pipelines seamlessly.
• File suffix enhancement. You can now configure a file suffix, such as txt or json, for output files generated by Hadoop FS, Local FS, MapR FS, and the Amazon S3 destinations.
• JDBC Producer destination enhancement - You can now configure the transaction isolation level that the JDBC Producer destination uses to connect to the database. Previously, the destination used the default transaction isolation level configured for the database.
• Kudu destination enhancement - You can now configure the destination to perform one of the following write operations: insert, update, delete, or upsert.

• XML processing enhancement - You can now generate records from XML documents using simplified XPath expressions with origins that process XML data and the XML Parser processor. This enables reading records from deeper within XML documents.
• Consolidated data format properties - You now configure the data format and related properties on a new Data Format tab. Previously, data formats had individual configuration tabs, e.g., Avro, Delimited, Log.

  Related properties, such as Charset, Compression Format, and Ignore Control Characters now appear on the Data Format tab as well.

• Checksum generation for whole files - Destinations that stream whole files can now generate checksums for the files so you can confirm the accurate transmission of the file.

• Add labels to pipelines from the Home page - You can now add labels to multiple pipelines from the Data Collector Home page. Use labels to group similar pipelines. For example, you might want to group pipelines by database schema or by the test or production environment.
• Reset the origin for multiple pipelines from the Home page - You can now reset the origin for multiple pipelines at the same time from the Data Collector Home page.

• Metric rules and alerts enhancements - The gauge metric type can now provide alerts based on the number of input, output, or error records for the last processed batch.

• New file functions - You can use the following new file functions to work with file paths:
  • file:fileExtension(<filepath>) - Returns the file extension from a path.
  • file:fileName(<filepath>) - Returns a file name from a path.
  • file:parentPath(<filepath>) - Returns the parent path of the specified file or directory.
  • file:pathElement(<filepath>, <integer>) - Returns the portion of the file path specified by a positive or negative integer.
  • file:removeExtension(<filepath>) - Removes the file extension from a path.
• New pipeline functions - You can use the following new pipeline functions to determine information about a pipeline:
  • pipeline:name() - Returns the pipeline name.
  • pipeline:version() - Returns the pipeline version when the pipeline has been published to Dataflow Performance Manager (DPM).
• New time functions - You can use the following new time functions to transform datetime data:
  • time:extractLongFromDate(<Date object>, <string>) - Extracts a long value from a Date object, based on the specified date format.
  • time:extractStringFromDate(<Date object>, <string>) - Extracts a string value from a Date object, based on the specified date format.
  • time:millisecondsToDateTime(<long>) - Converts an epoch or UNIX time in milliseconds to a Date object.