Supported Systems and Versions

Data Collector supports working with a wide range of external systems. StreamSets tests to verify that Data Collector performs without issues when working with those systems.

The following tables list the systems that Data Collector supports and tests, and the stages that work with those systems.

Cloud Native

Data Collector supports the cloud native providers listed in the following table. StreamSets tests the listed stages on the specified environments.

Customers with an enterprise account can receive help with the listed stages on the tested environment.
Note: Some of the following supported and tested versions have been deprecated. For a list, see Deprecated Functionality.
Supported Cloud Provider Stages Tested Environment
Amazon Origins:
  • Amazon S3
  • Amazon SQS Consumer
  • Kinesis Consumer
Destinations:
  • Amazon S3
  • Kinesis Firehose
  • Kinesis Producer
Executor:
  • Amazon S3

Credential Store:

  • Amazon Secrets Manager

 AWS
Databricks (Runtime 6.x, 7.x, or 8.x) Databricks Delta Lake destination

Databricks Job Launcher executor

Databricks Query executor

 Databricks Delta Lake Runtime 6.x, 7.x, or 8.x
Google Cloud Storage Origins:
  • Google BigQuery
  • Google Cloud Storage
  • Google Pub/Sub Subscriber
Destinations:
  • Google BigQuery (Legacy)
  • Google BigQuery (Enterprise)
  • Google Cloud Storage
  • Google Pub/Sub Subscriber
Executor:
  • Google BigQuery (Enterprise)
  • Google Cloud Storage
Credential Store:
  • Google Secret Manager
Google Cloud Storage
Microsoft Azure Origins:
  • Azure Data Lake Storage Gen1
  • Azure Data Lake Storage Gen2
  • Azure IoT/Event Hub Consumer
Destinations:
  • Azure Data Lake Storage Gen1
  • Azure Data Lake Storage Gen2
  • Azure Event Hub Producer
  • Azure IoT Hub Producer
  • Azure Synapse SQL
Executors:
  • ADLS Gen1 File Metadata
  • ADLS Gen2 File Metadata
Credential Store:
  • Azure Key Vault
 Microsoft Azure
MongoDB Atlas MongoDB Atlas origin

MongoDB Atlas destination

 MongoDB Atlas
Salesforce Origins:
  • Salesforce
  • Salesforce Bulk API 2.0
Processors:
  • Salesforce Lookup
  • Salesforce Bulk API 2.0 Lookup
Destinations:
  • Salesforce
  • Salesforce Bulk API 2.0
  • Tableau CRM
Salesforce
Snowflake Snowflake destination

Snowflake File Uploader destination

Snowflake executor

 Amazon S3

Microsoft Azure

Protocols

Data Collector supports the protocols listed in the following table. StreamSets tests the listed stages on the specified environments.

Customers with an enterprise account can receive help with the following protocols unless the implementation proves below standard for the protocol.

Private extensions for the protocols are not supported unless specified in the table.
Supported Protocol Stages Tested Environment
CoAP CoAP Server origin

CoAP Client destination

 Eclipse Californium 1.0.4
HTTP Origins:
  • HTTP Client
  • HTTP Server
  • NiFi HTTP Server
Processors:
  • HTTP Client
  • HTTP Router
Destinations:
  • HTTP Client
 Apache HTTP from Centos 6.8
JMS JMS Consumer origin

JMS Producer destination

 ActiveMq 5.14.3
MQTT MQTT Subscriber origin

MQTT Publisher destination

 Mosquitto
OPC UA OPC UA Client origin Full testing not performed at this time
SFTP/ FTP / FTPS SFTP/FTP/FTPS Client origin

SFTP/FTP/FTPS Client destination

SFTP/FTP/FTPS Client executor

 vsftpd 3.0
Syslog Syslog destination Full testing not performed at this time
TCP TCP Server origin Java TCP Stack
UDP UDP Multithreaded Source origin

UDP Source origin

 Java UDP Stack
Websocket Origins:
  • WebSocket Client
  • WebSocket Server
Destination:
  • WebSocket Client
 Java HTTP Stack

Versioned Systems

Versioned systems are external systems with multiple versions. When Data Collector supports multiple versions of an external system, you might need to install a specific stage library to work with a particular version, depending on your Data Collector installation. For details on individual stage libraries and the stages that they include, see Available Stage Libraries.

The following table lists the system versions that are supported and tested for Data Collector.

The supported versions column lists the system versions that customers with an enterprise account can receive help with. The tested versions column lists the subset of the supported versions that have been fully tested.

Note: Some of the following supported and tested versions have been deprecated. For a list, see Deprecated Functionality.
System Stages Supported Versions Tested Versions
Aerospike Aerospike destination Aerospike 3.15.x Full testing not performed at this time
Cassandra Cassandra destination Cassandra 1.2, 2.x, 3.x Cassandra 3.11
Couchbase Server Couchbase destination Couchbase Server 5.x Couchbase Server 5.1.1
Elasticsearch Elasticsearch origin

Elasticsearch destination

 Elasticsearch 5.x - 8.x Elasticsearch 5.20, 6.8.12, 7.9.0, 8.1.1
Flume Flume destination
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • CDP Private Cloud Base 7.1.x
Greenplum GPSS Producer destination Greenplum 5.x Greenplum 5.12.0
Hadoop Distributed File System (HDFS):

Data Collector cluster mode
Origin:
  • Hadoop FS
Destination:
  • Hadoop FS
Executors:
  • HDFS File Metadata
  • MapReduce
  • Amazon EMR 5.14.x with Hadoop 2.8.3.
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.x
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
Hadoop Distributed File System (HDFS):

Data Collector standalone mode
Origin:
  • Hadoop FS Standalone
Destination:
  • Hadoop FS
Executors:
  • HDFS File Metadata
  • MapReduce
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.x
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
Hashicorp Vault Hashicorp Vault credential store General support Full testing not performed at this time
HBase HBase Lookup processor
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.x
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
Hive Hive Metadata processor

Hive Metastore destination

Hive Query executor
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.x distribution of Hive 2.1
  • HDP 2.6.x distribution of Hive 1.x.
  • HDP 3.1.x
  • MapR 6.0.0 with MEP 4.x
  • MapR 6.0.1 with MEP 5.x
  • MapR 6.1.x with MEP 6.x
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
Hive Streaming Hive Streaming destination

Hive Query executor
  • Hive 0.13 and later
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • MapR 6.0.0 with MEP 4.x
  • MapR 6.0.1 with MEP 5.x
  • MapR 6.1.x with MEP 6.x
 Full testing not performed at this time
InfluxDB InfluxDB destination InfluxDB 0.9 - 1.x InfluxDB 0.13, 1.7.10
InfluxDB 2.x destination InfluxDB 2.x InfluxDB 2.0.8
Java Keystore Java Keystore credential store Java Virtual Machine Java Virtual Machine
Kafka:

Data Collector cluster mode

 Kafka Consumer origin
  • CDH 6.0.x - 6.3.x
  • CDH Kafka 3.1.x, 4.1.x with:
    • CDS powered by Spark 2.2 release 1
    • CDS powered by Spark 2.3 release 2, 3, 4
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.0
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • CDP Private Cloud Base 7.1.x
Kafka:

Data Collector standalone mode

 Origins:
  • Kafka Consumer
  • Kafka Multitopic Consumer
Destination:
  • Kafka Producer
  • Apache Kafka 1.0.x, 1.1.x, 2.0.x - 2.8.x, 3.0.x - 3.2.x
  • CDH 6.0.x - 6.3.x
  • CDH Kafka 3.1.x, 4.1.x
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.0
  • Apache Kafka 1.0.x, 1.1.x, 2.0.x - 2.8.x, 3.0.x - 3.2.x
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • CDH Kafka 2.1.0, 3.0.0, 3.1.0, 4.1.0
  • CDP Private Cloud Base 7.1.x

  • HDP 3.1.0
KineticaDB Kinetica destination
  • KineticaDB 6.0.x - 6.2.x
  • KineticaDB 7.0.x
 Full testing not performed at this time
Kudu Kudu Lookup processor

Kudu destination
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • CDP Private Cloud Base 7.1.x
MariaDB Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
 MariaDB 10.2 - 10.7 as limited drop-in replacements for MySQL 5.7.

For more information, see the MariaDB documentation.

 MariaDB 10.7
MapR DB Origin:
  • MapR DB
Destinations:
  • MapR DB
  • MapR DB JSON
  • MapR 6.0.0 with optional MEP 4.x
  • MapR 6.0.1 with optional MEP 5.x
  • MapR 6.1.x with optional MEP 6.x
  • MapR 6.0.0 with MEP 4
  • MapR 6.0.1 with MEP 5
  • MapR 6.1.x with MEP 6
MapR FS:

Data Collector cluster mode

 MapR FS origin

MapR FS destination
  • MapR 6.0.0 with MEP 4.x
  • MapR 6.0.1 with MEP 5.x
  • MapR 6.1.x with MEP 6.x
  • MapR 6.0.0 with MEP 4
  • MapR 6.0.1 with MEP 5
  • MapR 6.1.x with MEP 6
MapR FS:

Data Collector standalone mode

 MapR FS Standalone origin

MapR FS destination

MapReduce executor
  • MapR 6.0.0 with optional MEP 4.x
  • MapR 6.0.1 with optional MEP 5.x
  • MapR 6.1.x with optional MEP 6.x
  • MapR 6.0.0 with MEP 4
  • MapR 6.0.1 with MEP 5
  • MapR 6.1.x with MEP 6
MapR Streams Origins:
  • MapR Multitopic Streams Consumer
  • MapR Streams Consumer
  • MapR DB CDC
Destination:
  • MapR Streams Producer
 MapR 6.1.x with optional MEP 6.x MapR 6.1.x with MEP 6
MemSQL MemSQL Fast Loader destination MemSQL 6.8 and later MemSQL 6.8.15 with the MySQL Connector/J 8.0.12 driver
Microsoft SQL Server SQL Server 2019 BDC origin SQL Server 2019 Big Data Cluster SQL Server 2019 Big Data Cluster
SQL Server CDC Client origin

SQL Server Change Tracking origin
  • SQL Server 2017
  • SQL Server 2019
  • SQL Server 2017
  • SQL Server 2019
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
 SQL Server 2017 and later
  • SQL Server 2017
  • SQL Server 2019
MongoDB Origins:
  • MongoDB
  • MongoDB Atlas
Processor:
  • MongoDB Lookup
Destinations:
  • MongoDB
  • MongoDB Atlas
 MongoDB 3.x, 4.x MongoDB 3.6, 4.0
MongoDB Oplog origin MongoDB 3.x, 4.x MongoDB 3.6, 4.0
MySQL Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
 MySQL 5.7 and later
  • MySQL 5.7 with the MySQL Connector/J 8.0.12 driver
  • MySQL 8.0 with the MySQL Connector/J 8.0.12 driver
MySQL Binary Log MySQL 5.7 and later
  • MySQL 5.7 with the MySQL Connector/J 8.0.12 driver
  • MySQL 8.0 with the MySQL Connector/J 8.0.12 driver
NiFi NiFi HTTP Server origin General support Full testing not performed at this time
Omniture Omniture origin General support Full testing not performed at this time
Oracle Oracle Bulkload origin
  • Oracle 11g, 12c, 18c, 19c

Hosted systems and derived systems are not supported.
  • Oracle 11g, 19c with the Oracle 12.2.0.1.0 JDBC driver version
Oracle CDC Client origin
  • Oracle 11g, 12c, 18c, 19c, 21c
  • Oracle Real Application Clusters (RAC) 12c, 18c, 19c, 21c
  • Oracle Exadata appliances that run supported versions of Oracle RAC

Hosted systems and derived systems are not supported unless listed by name above.

  • Oracle 12c, 19c, 21c with the Oracle 12.2.0.1.0 JDBC driver version
  • Oracle RAC 12c, 19c with the Oracle 12.2.0.1.0 JDBC driver version
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
  • Oracle 11g, 12c, 18c, 19c, and later
  • Oracle Real Application Clusters (RAC) 12c, 18c, 19c, and later
Also supported:
  • Hosted systems, such as AWS RDS, that run supported versions of Oracle RAC
  • Derived systems, such as Oracle Exadata, that run supported versions of Oracle RAC

  • Oracle 11g, 19c with the Oracle 12.2.0.1.0 JDBC driver version

  • Oracle RAC 12c, 19c with the Oracle 12.2.0.1.0 JDBC driver version
PMML PMML Evaluator processor General support Full testing not performed at this time
PostgreSQL Aurora PostgreSQL CDC Client origin
  • Aurora PostgreSQL 2.2.0 (with PostgreSQL 10.6 - 10.17) and later
  • Aurora PostgreSQL 3 (with PostgreSQL 11.0 - 11.12)
  • Aurora PostgreSQL 4 (with PostgreSQL 12.0 - 12.7)
 Aurora PostgreSQL 4 (with PostgreSQL 12.7)
Origins:
  • JDBC Multitable Consumer
  • JDBC Query Consumer
Processors:
  • JDBC Lookup
  • JDBC Tee
Destination:
  • JDBC Producer
Executor:
  • JDBC Query
 PostgreSQL 9.x and later
  • PostgreSQL 9.6.9
  • PostgreSQL 10.4
  • PostgreSQL 11.7
  • PostgreSQL 12.2
  • PostgreSQL 13.0
  • PostgreSQL 14.0
PostgreSQL CDC Client origin
  • PostgreSQL 9.4 or later 9.x
  • PostgreSQL 10.x -13.x
  • PostgreSQL 9.6.9
  • PostgreSQL 10.4
  • PostgreSQL 11.7
  • PostgreSQL 12.2
  • PostgreSQL 13.0
  • PostgreSQL 14.0
Pulsar Origins:
  • Pulsar Consumer
  • Pulsar Consumer (Legacy)
Destination:
  • Pulsar Producer
Pulsar 2.x
  • Pulsar 2.1.0
  • Pulsar 2.2.1
  • Pulsar 2.3.2
  • Pulsar 2.4.2
  • Pulsar 2.5.1
  • Pulsar 2.6.2
RabbitMQ RabbitMQ Consumer origin

RabbitMQ Producer destination

 RabbitMQ 3.5.x and later RabbitMQ 3.5.6, 3.8.0
Redis Redis Consumer origin

Redis destination

 Redis 2.x - 4.x Redis 4.0.1
SAP HANA SAP HANA Query Consumer origin SAP HANA 2.4.x SAP HANA 2.0 with the SAP HANA JDBC driver version 2.4.76
Solr Solr destination
  • Apache Solr 6.x
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDP Private Cloud Base 7.1.x
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
Spark

Spark Evaluator processor

Spark executor
  • CDH 5.2.x - 5.16.x
  • CDH 6.0.x - 6.3.x
  • CDH Spark 2.1.x Release 1
  • CDP Private Cloud Base 7.1.x
  • HDP 3.1.x
  • CDH 6.0.1, 6.1.1, 6.2.0, 6.3.0
  • CDP Private Cloud Base 7.1.x
  • HDP 2.6.4.0, 3.1.0
Splunk Splunk destination General support Full testing not performed at this time
TensorFlow TensorFlow Evaluator processor TensorFlow 1.x Full testing not performed at this time
Teradata Teradata Consumer origin Teradata 16.x and later Teradata Database release 16.20 with the Teradata JDBC driver version 16.20.00.08
Thycotic Secret Server Thycotic Secret Server credential store Full testing not performed at this time Full testing not performed at this time