Runtime Values

Runtime values are values that you define outside of the pipeline and use for stage and pipeline properties. You can change the values for each pipeline run without having to edit the pipeline.

You can use runtime values for any pipeline property that allows the use of the expression language. You can, for example, use runtime values to represent batch sizes, timeouts, directories, and URI. You cannot use runtime values to represent fields.

You can use the following methods of passing runtime values to pipelines:

Runtime parameters: Use runtime parameters when you want to specify the values for pipeline properties when you start the pipeline.; You define runtime parameters when you configure the pipeline, and then you call the parameters from within that pipeline. When you start the pipeline, you specify the parameter values to use.; Runtime parameters are defined for a single pipeline - only that pipeline can call them.
Note: In Data Collector versions earlier than 2.5.0.0, pipeline runtime parameters were named pipeline constants.
Runtime properties: Use runtime properties when you want to define values for multiple pipeline properties in a file.; You define runtime properties in a file local to the Data Collector, and then you call the properties from within a pipeline. At runtime, Data Collector loads the property values from the file. A runtime properties file can contain multiple properties.; Runtime properties are defined for the entire Data Collector - any pipeline can call them.
Runtime resources: Use runtime resources when you want to secure sensitive information in files with restricted permissions.; You define runtime resources in a file local to the Data Collector, and then you call the resources from within a pipeline. You can restrict the permissions for the resource files to secure sensitive information. A resource file must contain one piece of information.; Runtime resources are defined for the entire Data Collector - any pipeline can call them.

Note: When configuring runtime values for pipelines run by StreamSets Control Hub, all Data Collector that run the pipeline must have the values locally defined in the expected locations.

Using Runtime Parameters

Runtime parameters are parameters that you define in a pipeline and then call from within that same pipeline. When you start the pipeline, you specify the parameter values to use. Use runtime parameters to specify values for pipeline properties when you start the pipeline.

Use runtime parameters to define values for stage and pipeline properties. For example, you can define an error directory parameter that points to different directories on a test machine and a production machine. Or you can define a connection parameter that points to different database connections for an origin in the test and production environments.

When you define a runtime parameter, you enter the default value to use. When you start the pipeline, you can specify another value to override the default. When the pipeline runs, the value replaces the name of the runtime parameter.

Note: If you shut down and then restart the Data Collector without stopping the pipeline, the pipeline continues running with the last set parameter values.

To implement runtime parameters, perform the following steps:

Define runtime parameters.
Use an expression in the pipeline to call a runtime parameter.
Start the pipeline with parameter values.

When you monitor a pipeline started with parameters, you can view the parameter values that the pipeline is currently using.

Step 1. Define Runtime Parameters

Define runtime parameters when you configure the pipeline.

In the pipeline properties, click the Parameters tab.
Using simple or bulk edit mode, click the Add icon and define the name and the default value for each parameter.
For example, define a parameter named JDBCConnectionString with the default value of jdbc:mysql://localhost:3306/sample.

Step 2. Call the Runtime Parameter

Use an expression to call a runtime parameter. You can use runtime parameters to represent any stage or pipeline property that allows the use of the expression language.

To call a runtime parameter in a stage or pipeline property, use the following syntax:

${<parameter name>}

For example, to use the JDBCConnectionString runtime parameter for the JDBC Multitable Consumer origin, enter the following syntax for the JDBC Connection String property:

${JDBCConnectionString}

You can call a runtime parameter from within an expression language function by simply entering the parameter name. For example, the following expression returns the value of the JDBCConnectionString runtime parameter:

 ${record:value(JDBCConnectionString)}

You can use a runtime parameter to represent a part of a property. For example, you could use a RootDir runtime parameter and append the rest of the directory in the property as follows:

${RootDir}/logfiles

You can also call a runtime parameter in the code developed for a scripting processor. The method you use to call the runtime parameter depends on the following scripting processor types:

JavaScript Evaluator or Jython Evaluator processor

Use the following syntax in any of the processor scripts: ${<parameter name>}. For example, the following line of JavaScript code assigns the value of the NewFieldValue parameter to a map field:

records[i].value.V= ${NewFieldValue}

Groovy Evaluator processor

Use the sdcFunctions.pipelineParameters() method in any of the processor scripts to return a map of all runtime parameters defined for the pipeline. For example, the following line of Groovy code assigns the value of the CompanyParam parameter to the Company Name field:

record.value['Company Name'] = sdcFunctions.pipelineParameters()['CompanyParam']

Step 3. Start the Pipeline with Parameters

When you start the pipeline, specify the values to use for the parameters.

Note: If you want to use the default parameter values, you can simply click the Start icon to start the pipeline.

From the pipeline canvas, click the More icon, and then click Start with Parameters.
If Start with Parameters is not enabled, the pipeline is not valid.
The Start with Parameters dialog box lists all parameters defined for the pipeline and their default values.
Override any default values with the values you want to use for this pipeline run.
Click Start.

Viewing Runtime Parameters

When you monitor a pipeline started with parameters, you can view the parameter values that the pipeline is currently using.

Before you can view the current runtime parameter values, verify that the monitoring settings include the Runtime Parameters chart.

In the Summary tab of the Monitor panel, click the More icon, and then click Settings.
In the Monitoring Settings dialog box, click in the Charts field and then select Runtime Parameters.
Click Save.
When you monitor a pipeline that you started with parameter values, the Runtime Parameters chart displays as follows:

The Runtime Parameters chart does not display when you start a pipeline without parameters.

Using Runtime Properties

Runtime properties are properties that you define in a file local to the Data Collector and call from within a pipeline. With runtime properties, you can define different sets of values for different Data Collectors.

Use runtime properties to define values for stage and pipeline properties. For example, you can define an error directory runtime property that points to different directories on a test machine and a production machine. Similarly, you might create test and production runtime properties for the origin and destination stages.

When defining a runtime property, you can use a static value or an environment variable.

When calling the runtime property, you can use it as part of a larger property definition. For example, you can set a runtime property to the HOME environment variable, which would differ on different machines, and then use the runtime property as a base directory for a longer directory.

To implement runtime properties, perform the following steps:

Define runtime properties.
Use an expression in the pipeline to call a runtime property.

Step 1. Define Runtime Properties

You can define runtime properties in the Data Collector configuration file, sdc.properties, or in a separate runtime properties file.

If you define the properties in a separate runtime properties file, use the required procedure for your installation type.

Data Collector configuration file

Use the following steps to define runtime properties in the Data Collector configuration file:

In the $SDC_CONF/sdc.properties file, configure the runtime.conf.location property as follows:
```
runtime.conf.location=embedded
```
To define the runtime properties in the $SDC_CONF/sdc.properties file, use either of the following formats:
- To use a value for a runtime property, use the following format:
```
runtime.conf_<property name>=<value>
```
  For example, the following runtime property defines a Hadoop FS directory template:
```
runtime.conf_HDFSDirTemplate=/HDFS/DirectoryTemplate
```
- To use an environment variable for a runtime property, use the following format:
```
runtime.conf_<property name>=${env("<environment_variable>")}
```
  For example, the following runtime property defines a base directory, setting it to the HOME environment variable:
```
runtime.conf_BaseDir=${env("HOME")}
```
Restart Data Collector to enable the changes.

Separate runtime properties file for RPM and tarball

Use the following steps to define runtime properties in a separate runtime properties file for an RPM or tarball installation:

Create a text file and save it in a directory relative to the $SDC_CONF directory.
To define runtime properties in a separate text file, use either of the following formats:
- To use a value for a runtime property, use the following format:
```
<property name>=<value>
```
  For example, the following runtime property defines a Hadoop FS directory template:
```
HDFSDirTemplate=/HDFS/DirectoryTemplate
```
- To use an environment variable for a runtime property, use the following format:
```
<property name>=${env("<environment_variable>")}
```
  For example, the following runtime property defines a base directory, setting it to the HOME environment variable:
```
BaseDir=${env("HOME")}
```
In the Data Collector configuration file, $SDC_CONF/sdc.properties, configure the runtime.conf.location property to point to the relative location of the separate runtime properties file.
For example, the following separate runtime properties file is located in a runtime directory that is relative to the $SDC_CONF directory:
```
runtime.conf.location=runtime/test-runtime.properties
```
Restart Data Collector to enable the changes.

Separate runtime properties file for Cloudera Manager

Use the following steps to define runtime properties in a separate runtime properties file for a Cloudera Manager installation:

Create a text file and define the runtime properties in the text file using either of the following formats:
- To use a value for a runtime property, use the following format:
```
<property name>=<value>
```
  For example, the following runtime property defines a Hadoop FS directory template:
```
HDFSDirTemplate=/HDFS/DirectoryTemplate
```
- To use an environment variable for a runtime property, use the following format:
```
<property name>=${env("<environment_variable>")}
```
  For example, the following runtime property defines a base directory, setting it to the HOME environment variable:
```
BaseDir=${env("HOME")}
```
Save the text file in the same directory on every node that runs Data Collector.
In Cloudera Manager, select the StreamSets service and click Configuration.
On the Configuration page, in the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc-env.sh field, add the following line to define the runtime properties file directory:
```
ln -sf /<directory>/runtime.properties "${CONF_DIR}/runtime.properties"
```
For example:
```
ln -sf /opt/sdc-runtime/runtime.properties "${CONF_DIR}/runtime.properties"
```
In the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc.properties field, configure the runtime.conf.location property to point to the separate runtime properties file by adding the following line:
```
runtime.conf.location=runtime.properties 
```
Restart Data Collector to enable the changes.

For more information, see Configuring Data Collector.

Step 2. Call the Runtime Property

Use the runtime:conf function to call a runtime property. You can use runtime properties to represent any stage or pipeline property that allows the use of the expression language.

To call a runtime property, use the following syntax:

${runtime:conf(<property name>)}

Note: If you defined the runtime properties in the Data Collector configuration file, enter just <property name> and not runtime.conf_<property name>.

For example, to use the HDFSDirTemplate runtime property for the Hadoop FS destination, enter the following syntax for the Directory Template property:

${runtime:conf('HDFSDirTemplate')}

You can use a runtime property to represent a part of a property. For example, you could use a RootDir runtime property and append the rest of the directory in the property as follows:

${runtime:conf('RootDir')}/logfiles

Using Runtime Resources

Similar to runtime properties, runtime resources are values that you define in a file local to the Data Collector and call from within a pipeline. But with runtime resources, you can restrict the permissions for the files to secure sensitive information. Use runtime resources to load sensitive information from files at runtime.

Use runtime resources to define sensitive values for stage and pipeline properties. You can, for example, use runtime resources to represent user names and passwords, or OAuth authentication information.

Tip: To more securely define sensitive values, use credential stores.

To implement runtime resources, perform the following steps:

Define each runtime resource.
Use an expression in the pipeline to call a runtime resource.

Step 1. Define Runtime Resources

Use the following steps to define runtime resources:

For each resource, create a text file and save it in the $SDC_RESOURCES directory.
A file must contain one piece of information to be used when the resource is called.
Optionally, restrict the permissions for the file.
Generally, anyone can read a file. To restrict permissions, configure the file so only the owner has read or write permissions for the file - in octals, that's 600 or 400. And the owner must be the system user that runs the Data Collector.

When you use the resource in the pipeline, you specify whether the file is restricted.

Step 2. Call the Runtime Resource

Use the runtime:loadResource or runtime:loadResourceRaw function to call a runtime resource. You can use runtime resources to represent sensitive information in any stage or pipeline property that allows the use of the expression language.

Note: In most cases, you'll use the runtime:loadResource function which trims any leading or trailing whitespace characters from the file. However, if needed, you can also use the runtime:loadResourceRaw function which includes any leading or trailing whitespace characters in the file.

To call a runtime resource, use the following syntax:

${runtime:loadResource(<file name>, <restricted: true | false>)}

For example, the following expression returns the contents of the JDBCpassword.txt file, trimming any leading or trailing whitespace characters. The file contains a password and is restricted so only the owner can read the file:

${runtime:loadResource("JDBCpassword.txt", true)}