Snowflake

You can create the following types of Snowflake connections:

Snowflake Connection

Available when using an authoring Data Collector version 3.20.0 or later that has the Snowflake Enterprise stage library version 1.7.x or later installed.

To create a Snowflake connection, the Snowflake Enterprise stage library, streamsets-datacollector-snowflake-lib, must be installed on the selected authoring Data Collector.

For a description of the Snowflake connection properties, see Snowflake Connection Properties.

After you create a Snowflake connection, you can use the connection in the following stages:
Engine Stage
Data Collector 3.20.0 or later
  • Snowflake destination
  • Snowflake File Uploader destination
  • Snowflake executor

Transformer 3.18.0 or later
  • Snowflake origin
  • Snowflake Lookup processor
  • Snowflake destination

Snowflake Roles and Privileges

When you configure a Snowflake stage, the Snowflake user that you specify in the stage must have a Snowflake role that includes all of the privileges required to perform the necessary Snowflake tasks.

Before using a connection in a Snowflake stage, ensure that the Snowflake user has the appropriate privileges. All Snowflake stages require specific privileges, except for the Data Collector Snowflake File Uploader destination.

For details about privilege requirements, see the documentation for the stage:
If you create a custom role with the required privileges, define the role that the stage uses in one of the following ways:
Assign the custom role as the default role
In Snowflake, assign the custom role as the default role for the Snowflake user account specified in the stage. A Snowflake user account is associated with a single default role.
Override the default role with the custom role
In the stage, select Use Snowflake Role on the connection information tab. Then, specify the name of the custom role. The custom role overrides the default role assigned to the Snowflake user account specified in the stage.
For example, you might define custom roles in Snowflake for specific data sources, and then specify one of the roles when using a Snowflake destination to load data.

Snowflake Connection Properties

When creating a Snowflake connection, configure the following properties on the Snowflake Connection Info tab:
Snowflake Connection Info Property Description
Snowflake Region Region where the Snowflake warehouse is located. Select one of the following regions:
  • An available Snowflake region.
  • Other - Use to specify a Snowflake region not listed in the property.
  • Custom JDBC URL - Use to specify a virtual private Snowflake installation.
Custom Snowflake Region Custom Snowflake region. Available when using Other as the Snowflake region.
Custom Snowflake URL Custom JDBC URL to use when using a virtual private Snowflake installation.
Account Snowflake account name.
User Snowflake user name.

The user account or the custom role that overrides the default role for this user account must have the required Snowflake privileges.

The required privileges depend on the stage where you use the connection. For details, see Snowflake Roles and Privileges.
Password Snowflake password.
Use Snowflake Role Overrides the default role for the specified user account.

The custom role must have the required Snowflake privileges, which are dependent on the stage where you use the connection. For details, see Snowflake Roles and Privileges.
Snowflake Role Name Name of the custom Snowflake role to use.

Snowpipe Connection

Available when using an authoring Data Collector version 3.20.0 or later that has the Snowflake Enterprise stage library version 1.7.x or later installed.

To create a Snowpipe connection, the Snowflake Enterprise stage library, streamsets-datacollector-snowflake-lib, must be installed on the selected authoring Data Collector.

For a description of the Snowpipe connection properties, see Snowpipe Connection Properties.

After you create a Snowpipe connection, you can use the connection in the following stage:
Engine Stage
Data Collector 3.20.0 or later Snowflake destination

Snowpipe Connection Properties

Define a Snowpipe connection when configuring the Data Collector Snowflake destination to use the Snowpipe load method. Before creating a Snowpipe connection, make sure to complete the Snowpipe prerequisite tasks.

When creating a Snowpipe connection, configure the following properties on the Snowpipe tab:

Snowpipe Property Description
Private Key PEM Private key PEM. Generated in Snowflake as part of the Snowpipe prerequisite tasks.
Tip: To secure sensitive information, you can use credential stores or runtime resources.
Private Key Password Private key password. Generated in Snowflake as part of the Snowpipe prerequisite tasks.
Public Key PEM Public key PEM. Generated in Snowflake as part of the Snowpipe prerequisite tasks.
Use Custom Snowpipe Endpoint Enables using a custom Snowpipe endpoint.
Use TLS Enables the use of TLS to connect to the custom Snowpipe endpoint.
Custom Snowpipe Host Host name for the custom Snowpipe endpoint.
Custom Snowpipe Port Port number for the custom Snowpipe endpoint.